\n<\/p>\n
Data were acquired from two adult male macaques (Macaca mulatta<\/i>, average weight of 17\u2009kg (subject 1 (S1)) and 10\u2009kg (S2), average age of 9\u2009years (S1) and 7\u2009years (S2)). This sample size was chosen to match the standard for neural recording studies of behaviour in monkeys22<\/a>,24<\/a><\/sup>. All animal procedures complied with the NIH Guide for the Care and Use of Laboratory Animals and were approved by the Institutional Animal Care and Use Committee of the Rockefeller University (protocol 24066-H).<\/p>\n After undergoing initial task training in their home cages, the subjects underwent two surgeries: the first to implant an acrylic head implant with a headpost and the second to implant electrode arrays. Both surgeries followed standard protocols, including for anaesthetic, aseptic and postoperative treatments. In the first surgery, a custom-designed MR-compatible Ultem headpost was implanted, surrounded by a bone cement cranial implant, or a headcap (Metabond, Parkell and Palacos, Heraeus), which was secured to the skull using MR-compatible ceramic screws (Rogue Research). After a 6-month interval, to allow bone to grow around the screws and for the subject to acclimate to performing the task during head fixation via the headpost, we performed a second surgery to implant 16 floating microelectrode arrays (32-channel FMA, Microprobes for Life Science) using standard procedures69<\/a><\/sup>. In brief, after performing a craniotomy and durotomy over the target area, arrays were inserted one by one stereotactically while\u00a0held at the end of a stereotaxic arm with a vacuum suction attachment (Microprobes). Using vacuum suction enabled us to release the arrays, after insertion, with minimal mechanical perturbation by turning off the suction. After all arrays had been implanted, the dura mater was loosely sutured and covered with DuraGen (Integra LifeSciences). The craniotomy was closed with bone cement.<\/p>\n We used standard density arrays (1.8\u2009mm\u2009\u00d7\u20094\u2009mm) for all areas, except SMA and preSMA, for which we used four high-density arrays (1.6\u2009mm\u2009\u00d7\u20092.95\u2009mm). Four additional electrodes on each array served as the reference (two electrodes)\u00a0and the ground\u00a0(two electrodes). Two arrays each were targeted to multiple areas of the frontal cortex, with locations identified stereotactically, and planned using brain surface reconstructions derived from anatomical MRI scans (3D Slicer 5.6.2). Locations were selected on the basis of their published functional and anatomical properties (see below), anatomical sulcal landmarks and a standard macaque brain atlas70<\/a><\/sup>. During surgery, locations were further adjusted on the basis of cortical landmarks and to avoid visible blood vessels. Arrays were implanted in the right hemisphere (contralateral to the arm used for behaviour).<\/p>\n Array locations are depicted in Fig. 3<\/a> and Extended Data Fig. 6<\/a> and confirmed with intraoperative photographs. For M1, we targeted hand and arm representations (F1) directly medial to the bend of the central sulcus (which corresponds roughly to the intersection of the central sulcus and the arcuate spur if the latter was extended caudally), based on retrograde labelling from the spinal cord and microstimulation of M171<\/a><\/sup> and M1 recordings72<\/a><\/sup>. For\u00a0PMd, we placed both arrays lateral to the precentral dimple, with one (more caudal) array directly medial to the arcuate spur (the arm representation71<\/a>,72<\/a>,73<\/a><\/sup>, F2), and the other more rostral (straddling F2 and F7). For\u00a0PMv, we targeted areas caudal to the inferior arm of the arcuate sulcus (F5), which are associated with hand movements based on retrograde labelling from the spinal cord71<\/a><\/sup> and M174<\/a><\/sup>, microstimulation75<\/a><\/sup> and functional studies51<\/a>,76<\/a>,77<\/a><\/sup> and with decision making77<\/a>,78<\/a><\/sup>. These areas contain neurons interconnected with PFC74<\/a><\/sup>. For SMA (F3) and preSMA (F6), we targeted the medial wall of the hemisphere, with the boundary between SMA and preSMA defined as the anterior\u2013posterior location of the genu of the arcuate sulcus, consistent with previous studies finding differences across this boundary in anatomical connectivity (for example, direct spinal projections in SMA but not preSMA79<\/a><\/sup>) and function42<\/a>,80<\/a><\/sup>. SMA arrays were largely in the arm representation79<\/a><\/sup>. For dlPFC, we targeted the region immediately dorsal to the principal sulcus (46d), following previous studies of action sequencing27<\/a>,28<\/a>,81<\/a><\/sup> and other cognitive functions82<\/a><\/sup>. For vlPFC, we targeted the inferior convexity ventral to the principal sulcus, with one (more rostral) array directly ventral to the principal sulcus (46v) and the other rostral to the inferior arm of the arcuate sulcus (45A\/B) based on evidence that encoding of abstract concepts occurs in regions that broadly span these two locations29<\/a>,83<\/a>,84<\/a><\/sup>, including a possibly heightened role (compared with dlPFC) in encoding abstract concepts in a manner invariant to temporal or spatial parameters84<\/a>,85<\/a>,86<\/a><\/sup>. For FP, we targeted a rostral location similar to previous recording and imaging studies (one array fully in area\u200910, the other straddling areas 9\u2009and 10)87<\/a>,88<\/a><\/sup>, including areas associated with executive functions89<\/a><\/sup>. In general, array locations targeted the cortical convexity immediately next to sulci, instead of in the banks, to allow shorter insertion depths that minimize the risk of missing the target\u00a0or damaging blood vessels. The exceptions were SMA and preSMA in the medial wall, for which this was not possible. To avoid damaging the superior sagittal sinus, we positioned the arrays laterally (2\u2009mm from midline) and slanted the electrodes medially (Extended Data Fig. 6<\/a>).<\/p>\n The lengths of each electrode were custom designed to target half-way through the grey matter and to substantially vary across the array to maximize sampling of the cortical depth. The following electrode lengths were used (in mm): 1.5\u22123.5 (M1), 1.5\u22123.1 (PMd and PMv), 2.8\u22125.8 (SMA and preSMA), 1.5\u22122.5 (dlPFC and vlPFC) and 1.5\u22122.6 (FP) for S1; and 1.7\u22123.75 (M1), 1.5\u22123.3 (PMv), 1.5\u22123.1 (PMd), 2.65\u22125.95 (SMA and preSMA), 1.75\u22123.15 (dlPFC), 1.35\u22123.2 (vlPFC) and 1.6\u22122.9 (FP) for S2. Reference electrodes were longer (6\u2009mm) to anchor the arrays. All electrodeswere Pt\/Ir (0.5\u2009M\u03a9), except 4, which were\u00a0Ir (10\u2009k\u03a9). Array connectors (Omnetics, A79022) were housed in custom-made Ultem pedestals (Crist), which were secured with bone cement onto the cranial implant. Four pedestals were used per subject, holding 5, 5, 4 and 2 connectors each.<\/p>\n The subjects were seated comfortably in the dark with their head restrained by headpost fixation. They faced a touchscreen (Elo 1590L 15-inch E334335, PCAP, 768\u2009\u00d7\u20091,024 pixels, refresh rate of 60\u2009Hz, with a matte screen protector to reduce finger friction) that presented images and was drawn on. The touchscreen location was optimized to allow each subject to easily draw at all relevant locations on the screen (23\u201326\u2009cm away; diagram in Extended Data Fig. 1<\/a>). Both subjects decided on their own over the course of learning to perform the task with the left hand. The chairs were designed to minimize movements of the torso and legs (by using a loosely restricting \u2018belly plate\u2019) and the non-drawing arm (by resting on the belly plate and having movement restricted to within the chair). Gravity-delivered reward (water\u2013juice mixture) was controlled by the opening and closing of a solenoid pinch valve (Cole-Parmer, 1\/8-inch inner diameter). The subjects were water-regulated, with careful monitoring to ensure that consumption met the minimum requirement per day (typically exceeding it), and body weight was closely monitored to ensure good health. The task was controlled with custom-written software using the MonkeyLogic (v.2.2.45) behavioural control and data acquisition MATLAB package90<\/a><\/sup> (PC: Windows 10 Pro, Intel Core i7-4790K, 32GB RAM; DAQ: National Instruments PCIe-6343). All stimuli (images of line figures defined as point sets, with points rendered large enough to appear as continuous curves) were also generated with custom-written MATLAB (R2021a) code. Images were presented in a workspace area on the screen (16.6\u2009cm\u2009\u00d7\u200916.9\u2009cm, corresponding to approximately 37\u00b0 by 38\u00b0 visual angle). Shape components in images were on average 4.0\u2009cm (9\u00b0) (maximum of width and height).<\/p>\n Each recording session consisted of 2\u22123.5\u2009h of recording. We collected 5\u201320 trials per condition (that is, each unique image for Figs. 4<\/a> and 5<\/a> and the single-shape task in Fig. 6<\/a>, and each primitive stroke for the character task in Fig. 6<\/a>). All trials were shuffled across all conditions in the session and presented in a randomly interleaved fashion, except for one case, the experiment in Fig. 6<\/a>, in which character and single-shape tasks were switched in blocks.<\/p>\n Before surgery, the naive subjects underwent initial training on core task components (that is, to trace images accurately using a sequence of discrete strokes). Early training took place in the home cage using custom-built rigs that were attached to an opening in the cage using the same hardware and software described above, except for the computer (Lenovo IdeaPad 14-inch laptop, Windows 10, AMD Ryzen 5 3500U, 8GB RAM) and DAQ (National Instruments USB-6001). This initial training progressed through seven stages. (1) Touch circle. The subjects were rewarded for touching a circle anywhere within its bounds. The circle started large, filling the entire screen, and shrank over trials to enforce more accurate touches. (2) Touch with a single finger. We shrank the circle until it was so small that it could only be touched with a single finger. The trial aborted if the subject touched outside the circle or with multiple fingers simultaneously. (3) Hold still. The subjects were rewarded for keeping their fingertip still on a dot, with the duration of this hold increasing across trials for up to a few seconds. (4) Track moving dot. The subjects had to track the dot with their finger as it moved (a lag between dot and finger was allowed). (5) Trace a line. We increased the speed of the moving dot over trials until eventually the dot moved so fast that the line it traced immediately appeared. We then positioned the line at locations far from the hold position to train the subject to raise its finger from the hold position and to trace lines at arbitrary locations, angles and lengths. (6) Trace single shape. We presented shapes of increasing difficulty (gradually morphing across trials from a straight line), including arcs, L-shapes, squiggles and circles. Across these stages, the shapes were presented at random locations. We did not enforce any particular tracing trajectory for each shape, which allowed the subjects to choose on their own. (7) Trace multiple shapes. We presented images composed of multiple disconnected shapes. This trained the subjects to understand that they should use multiple strokes to trace multiple shapes. At this point, the subject understood the basic structure of the task: to trace shapes using multiple strokes if needed. The progression across these stages was not determined by strict quantitative criteria but instead on a combination of quantitative and qualitative evaluations of task performance.<\/p>\n The subjects then practised various tasks to incentivize the learning of stroke primitives (consistent stroke trajectories for each shape). They practised single-shape trials using the set of diverse simple shapes in Fig. 1e<\/a>, varying randomly in shape and location across trials. We chose this set of shapes to cover a range of trajectory profiles (by varying rotation, the number of direction changes and whether shapes were curved or linear) and yet were simple enough to draw with one stroke and to combine multiple (two to six) shapes into single-character images. We did not constrain the subjects to learn specific stroke trajectories for each primitive; therefore, differences between primitives reflected each subject\u2019s own learning trajectory\u00a0for how to draw each shape. Note that our study did not depend on using an optimal set of shapes or primitives but instead depended on learning primitives and then demonstrating behavioural generalization using these primitives, as we found for our subjects. For S2, the four S-shapes and four arc shapes were also sometimes presented as rectilinear versions. For S-shapes these resembled Z, and for arc these resembled squares missing one edge. S2 drew these rectilinear versions using curved strokes similar to those used for S and arc shapes. Therefore we combined data for each rectilinear shape with its respective curved version. On different days, the subjects also practised multishape and character tasks.<\/p>\n Trials (event sequence in Fig. 1c<\/a>, screen schematic in Extended Data Fig. 1c<\/a>) began when the subject pressed and held a finger fixation button (blue square) at the bottom of the screen (note that button always means a virtual button). After a random delay (uniform, earliest 0.4\u20130.6\u2009s and latest 0.8\u20131.0\u2009s across experiments for S1 and 0.8\u20131.1\u2009s for S2) the image appeared (dark grey on a light-grey background). After a random delay (uniform, ranging from 0.6\u20131.0\u2009s to 1.2\u20131.6\u2009s across experiments for S1, and 1.1\u20131.5\u2009s to 1.8\u20132.4\u2009s for S2), a go cue (400\u2009Hz tone and image blank for 300\u2009ms) was presented. During this delay between image presentation and the go cue (planning epoch), the finger had to be kept still on the fixation button, but the subject was free to look anywhere. After the go cue, the subject was free to move their hand towards the image and start drawing. Immediately after the finger was raised from the fixation button, it disappeared and a \u2018done button\u2019 (green square) appeared at its location\u00a0and stayed there. During drawing, the image stayed visible and the finger left a trail of black \u2018ink\u2019 on the screen. The subject signalled drawing completion by pressing the done button (effectively no time limit was imposed). This was followed by performance feedback, which spanned four modalities, each signalling performance: (1) screen colour, (2) sound, (3) duration of delay before getting reward (time out) and (4) reward. First, screen colour and sound were signalled, followed by the time out, and then the reward (detailed below). In addition to this feedback at the end of the trial, we also provided online feedback by immediately aborting the trial in case of serious errors; for example, touching far from any image points or for single-shape and multishape trials\u00a0(but not for character trials), using more than one stroke per shape component. These online abort modes were turned off for trials testing novel characters.<\/p>\n Screen image changes (including image presentation and other trial events) were recorded using photodiodes (Adafruit Light Sensor ALS-PT19), and sounds were recorded using an electret microphone (Adafruit Maxim MAX4466, 20-20KHz). We performed eye tracking (ISCAN), but did not enforce eye fixation.<\/p>\n Behaviour was scored by aggregating multiple metrics or factors. There were three classes of factors. The factors that had the greatest influence on the final aggregate score measured image similarity, or the similarity of the final drawing to the target image (ignoring its temporal trajectory). We also computed factors that reflected behavioural efficiency and, in some cases, factors that were task-specific. These scores were computed using behavioural data (a sequence of touched xy<\/i> coordinates with gaps between strokes) and image data (a set of xy<\/i> coordinates). Below, we describe the factors and then how they were aggregated into a single score.<\/p>\n This included two factors: drawing-image overlap and Hausdorff distance. Drawing-image overlap was the fraction of the image points that were touched (within a margin of error) by at least one of the drawn points. A subset of the image points were weighted more heavily because they captured characteristic features of the shape (for example, the corners and end points of an L-shape). Hausdorff distance is a metric used to measure the distance between the set of drawn points and the set of image points (definition below).<\/p>\n To incentivize efficiency, we included a factor that compares the cumulative distance travelled in the drawing (that is, the amount of ink) to the cumulative distance of the edges of the figure in the image, with its value negatively proportional to the excess of drawn ink over image ink.<\/p>\n During practice trials for the character task (see the section \u2018Task types\u2019), we also included factors that capture the extent to which drawn strokes matched the shapes used in the image. This included two factors: one proportional to the similarity of the number of strokes and the number of image shapes, and the other proportional to the spatial alignment of the drawn strokes to the image shapes. Importantly, these factors were included only for practice images and not for novel test images.<\/p>\n The final score aggregated the image similarity, behavioural efficiency and task-specific factors, with more weight on image similarity factors. We first rescaled factors linearly between 0 and 1 (where 1 means good performance), with the dynamic range set by a lower and upper bound. These bounds were adaptively updated on every trial based on the distribution of factor values in the last 50 trials (lower bound set to the 1st percentile and upper bound to the 53rd percentile), which ensured that the dynamic range of feedback matched the dynamic range of behavioural performance from recent history. We then weighted each factor to tune its relative contribution (using weights hand-tuned for each experiment; generally highest for image similarity) and computed the final scalar score (range 0 to 1) using the worst factor after weighting:<\/p>\n $${s}_{\\mathrm{scal}}=\\mathop{\\min }\\limits_{i}(1-{w}_{i}(1-{f}_{i}))$$<\/span><\/p>\n<\/div>\n where i<\/i> indexes the factors, w<\/i>i<\/i><\/sub> are the weights (between 0 and 1) and f<\/i>i<\/i><\/sub> are the factor values (between 0 and 1). We also gave each trial a categorical score: great (s<\/i>scal<\/sub>\u2009>\u20090.82), good (0.65\u2009<\u2009s<\/i>scal<\/sub>\u2009\u2264\u20090.82), OK (0.15\u2009<\u2009s<\/i>scal<\/sub>\u2009\u2264\u20090.65) or fail (s<\/i>scal<\/sub>\u2009\u2264\u20090.15).<\/p>\n The scalar and categorical scores determined the feedback across the four different modalities. The meaning of screen colour and sound were learned, whereas delay and reward had intrinsic value. For screen colour, a linear interpolation between two colours, such that a score of 0 was mapped to red (RGB: 1, 0.2, 0) and 1 was mapped to green (0.2, 1, 0.2). For sound cue, a sound was determined by s<\/i>scal<\/sub>: if great, then three pulses (1,300\u2009Hz, 0.16\u2009s on and off); if good, then a single pulse (1,000\u2009Hz, 0.4\u2009s); if OK, then no sound; if fail, then a single pulse (120\u2009Hz, 0.27\u2009s). For delay until reward, a nonlinear mapping was generated from score to delay before reward. We first applied a linear mapping from the scalar score, such that a score of 0 was mapped to a long delay (5\u2009s\u2009+\u2009a random uniform jitter of 0\u20132.5\u2009s), and a score of 1 was mapped to 0\u2009s of delay. Furthermore, if s<\/i>scal<\/sub> was great, good or OK, this delay was reduced by multiplying by 0.65. Finally, for reward, the open duration of the solenoid gating the juice line was defined as<\/p>\n $${\\rm{r}}{\\rm{e}}{\\rm{w}}{\\rm{a}}{\\rm{r}}{\\rm{d}}={C}\\times m\\times a\\times {s}_{{\\rm{s}}{\\rm{c}}{\\rm{a}}{\\rm{l}}}$$<\/span><\/p>\n<\/div>\n where C<\/i> is a constant in dimensions of time (0.15\u22120.6\u2009s, manually set depending on the difficulty of the task); m<\/i> is a multiplier that gives a bonus for good performance and further penalizes bad performance, depending on the value of s<\/i>scal,<\/sub> great (1.3), good (1.0), OK (0.8) or fail (0); a<\/i> is a random variable sampled from the uniform distribution a<\/i>\u2009~\u20090.75\u2009+\u20090.5\u2009\u00d7\u2009U<\/i>(0,1); and s<\/i>scal<\/sub> is defined as above. On average, including failed trials, the subjects received around 0.35\u2009ml reward per trial. The temporal\u00a0order in which these four feedback signals were delivered is described above (see the section \u2018Trial structure\u2019).<\/p>\n The single-shape task presented one of the practised simple shapes or, in the categories experiment, sometimes a morphed shape. The subjects were only allowed to use a single stroke (triggering online abort if more was used). In four single-shape sessions for S1, the ending of the drawing epoch was triggered by stroke completion (that is, on finger raise) not on the pressing of the done button as in all other sessions and experiments.<\/p>\n To test for motor invariance (Figs. 2a\u2013e<\/a> and 4<\/a>), we presented images of practised shapes, varying across trials in location, size or both. For location variation, images spanned 321\u2009pixels (9.6\u2009cm) in the x<\/i> and y<\/i>\u2009dimensions (measuring between shape centres), which is 2.38 times the average size of shapes (135\u2009pixels, 4.0\u2009cm, maximum across width and height). For size variation, the maximum size was 2.5 times larger than the smallest (in diameter), except for two experiments for S1, in which the ratio was 2.0. The location and size variation in Fig. 2b,c<\/a> is representative (S1, n<\/i>\u2009=\u20092 sessions varying in location, n<\/i>\u00a0=\u00a03 varying in size; S2, n<\/i>\u2009=\u20093 sessions varying in location, n<\/i>\u00a0=\u00a02 varying in size).<\/p>\n Size and location variation was chosen based on previous studies of M1 activity\u00a0and electromyography of muscles controlling the arm during reaching in macaques91<\/a>,92<\/a><\/sup>. For reaches performed along the coronal plane around 25\u2009cm from the subject, similar to the geometry of the touchscreen\u00a0relative to the subject in our study, M1 and electromyographic activity are\u00a0substantially affected by translating the\u00a0reach location by 10\u2009cm (ref. 91<\/a><\/sup>), similar to the location variation in our study (9.6\u2009cm), and by varying the scale of the reach by 2-fold (7\u2009cm to 12\u2009cm)92<\/a><\/sup>, similar to the size variation in our study (2\u20132.5-fold).<\/p>\n To test for categorical structure (Figs. 2f\u2013j<\/a> and 5<\/a>), we constructed morph sets (S1, n<\/i>\u2009=\u20097 morph sets across 3 sessions; S2, n<\/i>\u2009=\u200913 morph sets across 4 sessions), each consisting of two practised shapes and four to five images that morph between those shapes through linear interpolation along shape parameters, such as the extent of closure of the top of the U (Fig. 2g<\/a>). Across morph sets, we varied different image parameters (Extended Data Fig. 3<\/a>).<\/p>\n Each image was composed of two to four shapes positioned at random, nonoverlapping, locations spanning the space of the screen (possible locations include the four corners and the centre). The subjects were allowed to draw the shapes in any order and to use any trajectory for each shape, but were constrained to use one stroke per shape and to not trace in the gaps between shapes. We present results averaged across two sessions, one from each subject (Extended Data Fig. 4<\/a>). For each trial, an image was constructed by sampling shapes randomly without replacement. This led to n<\/i>\u2009=\u2009531 (S1) and n<\/i>\u2009=\u2009278 (S2) unique images.<\/p>\n Each image was generated by connecting two to six simple shapes into a single character by sampling from a generative model as follows. A character with N<\/i> shapes was defined by randomly sampling N<\/i> shapes and N<\/i>\u2009\u2013\u20091 relations, where each relation (indexed i<\/i>) defines the locations of the attachment points on shapes i<\/i> and i<\/i>\u2009+\u20091, which in turn define how the shapes connect to each other. This approach is similar to a previous generative model for handwritten characters9<\/a><\/sup>. Generated characters were only kept if there was minimal crossing of shapes over each other.<\/p>\n For experiments testing behavioural generalization to novel characters (Fig. 2k\u2013p<\/a>), we mixed practised and novel characters (practised, n<\/i>\u2009=\u2009189 (mean, range 22\u2013491) per day; novel, 48 (mean, range 0\u2013155) per day). For analyses, we labelled as \u2018novel\u2019 only the very first trial for a given character. Because of random sampling in generating characters, it would in principle be possible that characters generated on different days are in fact identical. To avoid this possibility, we ensured post hoc that all characters labelled novel were different from every previously encountered character across all days (quantified using the Hausdorff distance).<\/p>\n For neural experiments comparing single-shape and character tasks (Fig. 6<\/a>), we analysed the sessions for which we collected data from both the single-shape and character tasks (S1, n<\/i>\u2009=\u20099 sessions, median N<\/i> matching primitives between single-shape and character tasks\u00a0=\u20099 (range 5\u221212); S2, n<\/i>\u2009=\u20099 sessions, median N<\/i> matching primitives\u2009=\u200910 (range 2\u221214)). We switched between single-shape and character tasks using a block design (2\u22125 blocks each per session), except one session for S2, which used random interleaving across trials.<\/p>\n Touchscreen data were represented as time series of (x<\/i>, y<\/i>) coordinates in units of pixels (conversion: 33.6\u2009pixels per cm) and sampled at 60\u2009Hz, which we upsampled to 500\u2009Hz (performed in MonkeyLogic to align all behavioural signals, including trial event markers and eye tracking) and low-pass filtered to keep only drawing-related movements\u00a0(15\u2009Hz). Strokes were segmented based on the time of first touch (onset) and the time of last touch (offset) with 500\u2009Hz resolution.<\/p>\n For some analyses (Fig. 1f<\/a> and as input to the trajectory distance below) we further computed stroke instantaneous velocity and speed as follows. Extracted strokes were low-pass filtered (12.5\u2009Hz) and downsampled to 25\u2009Hz. We then used the standard five-point stencil method to compute a finite difference approximation of the derivative (separately for the x<\/i> and y<\/i> coordinates):<\/p>\n $${f}^{{\\prime} }[n]=\\frac{f[n-2]-8f[n-1]+8f[n+1]-f[n+2]}{12\\,{\\text{h}}}$$<\/span><\/p>\n<\/div>\n where f<\/i>[n<\/i>] is a discrete time series (that is, the x<\/i> or y<\/i> coordinates) indexed by integer n<\/i>, and h<\/i> is the sampling period in seconds. The resulting velocity time series was upsampled to the original 500\u2009Hz sampling rate with a cubic spline. Speed was computed as the norm of the (x<\/i>, y<\/i>) velocity at each time point.<\/p>\n To quantify the similarity between any two strokes based on their spatiotemporal trajectories while ignoring their relative size and location, we devised a trajectory distance metric, a scalar dissimilarity score based on the dynamic time warping distance between two strokes represented as velocity time series v<\/i>1<\/sub> and v<\/i>2<\/sub>. To compute trajectory distance between two strokes, we spatially rescaled each stroke (while maintaining its x<\/i>y<\/i> aspect ratio) to make the diagonal of its bounding box unit length\u20091. We then linearly interpolated each stroke to the same number of points (70) to allow point-by-point comparison between strokes. This was done spatially by interpolating based on fraction of cumulative distance travelled (so that the distances between successive points were the same over the entire stroke) to capture the spatiotemporal trajectory, as in a previous analyses of strokes in handwriting9<\/a><\/sup>. Interpolated trajectories were converted to velocity time series as described above. We then computed the dynamic time-warping distance between velocities v<\/i>1<\/sub> and v<\/i>2<\/sub>:<\/p>\n $${D}_{\\text{DTW}}({v}_{1},{v}_{2})=\\frac{{\\text{min}}_{{\\rm{\\pi }}}{\\sum }_{(i,j)\\in {\\rm{\\pi }}}d(i,j)}{{\\text{N}}}$$<\/span><\/p>\n<\/div>\n where i<\/i> and j<\/i> index the two velocity trajectories, N<\/i> is the number of points (70), and \u03c0 is a set of (i<\/i>,\u00a0j<\/i>) pairs representing a contiguous path from (0, 0) to (N<\/i>, N<\/i>). The local distance metric d<\/i>(i<\/i>,\u00a0j<\/i>) is the Euclidean distance plus a regularization factor to discourage excessive warping:<\/p>\n $$d(i,j)=|{v}_{1}[i]-{v}_{2}[\\,j]|+\\lambda \\times |i-j|,$$<\/span><\/p>\n<\/div>\n $$\\lambda =0.045{\\langle |{v}_{n}[i]|\\rangle }_{i,n}.$$<\/span><\/p>\n<\/div>\n For the regularization parameter, \u03bb<\/i>, the purpose of the summation term was to rescale it to match the magnitude of velocities. The resulting distance D<\/i>DTW<\/sub> was then rescaled between 0 and 1 to return the trajectory distance:<\/p>\n $${D}_{\\mathrm{traj}}({v}_{1},{v}_{2})=1-\\frac{1}{{D}_{\\mathrm{DTW}}({v}_{1},{v}_{2})+1}.$$<\/span><\/p>\n<\/div>\n To compare the similarity of two images\u2014each a set of (x<\/i>, y<\/i>) points\u2014we used a modified version of the Hausdorff distance, a distance metric commonly used in machine vision for comparing the similarity between two point sets based on shape attributes93<\/a><\/sup>. There are, in principle, at least 24 variants of the Hausdorff distance based on possible formula variations93<\/a><\/sup>. Here we used a variant that is minimally susceptible to outlier points because it takes means instead of minima and maxima (variant\u200923 in the referenced study93<\/a><\/sup>). Image distance was computed as follows: (1) each image was centred so its centre of mass was at (0, 0); (2) the image distance was then computed. First, we defined the distance between two points, d<\/i>(a<\/i>,\u00a0b<\/i>), as the Euclidean distance. We also defined the distance between a point and a set of points, d<\/i>(a<\/i>,\u00a0B<\/i>), and the distance from set A<\/i> to set B<\/i>, d<\/i>(A<\/i>,\u00a0B<\/i>), as follows:<\/p>\n $$d(a,B)=\\mathop{\\min }\\limits_{b\\in B}\\,d(a,b),$$<\/span><\/p>\n<\/div>\n $$d(A,B)=\\frac{1}{|A|}\\sum _{a\\in A}d(a,B).$$<\/span><\/p>\n<\/div>\n The image distance was then defined as:<\/p>\n $${D}_{\\mathrm{image}}(A,B)=\\frac{d(A,B)+d(B,A)}{2}.$$<\/span><\/p>\n<\/div>\n For experiments on categorical structure, we generated a set of images with each set containing four to five novel images that morph between one primitive (P1) and another primitive (P2) to create a morph set. Each trial presented a single image from one morph set. We sought to quantify the relative similarity between the data of a given trial\u2014its behavioural, image or neural data (see below)\u2014and data for the two primitives, P1 and P2, in its morph set. To do so, we devised a primitive alignment score defined as:<\/p>\n $$a=\\frac{{d}_{1}}{{d}_{1}+{d}_{2}}$$<\/span><\/p>\n<\/div>\n where d<\/i>1<\/sub> is the average of the distances between a given trial and each of the P1 trials, and d<\/i>2<\/sub> the average distance to the P2 trials. A score closer to 0 implies similarity to P1, and a score close to 1 implies similarity to P2 (note that\u00a0in practice the\u00a0primitive alignments for P1 and P2 data are not exactly 0 and 1 owing to trial-by-trial variation). The particular metric used for these distances depended on the analysis: image distance (for images), trajectory distance (for drawings) or Euclidean distance between population activity vectors (for neural activity). We confirmed that primitive alignment scores for image data varied linearly with morph number (Fig. 2j<\/a> and Extended Data Fig. 3c<\/a>), which ensured that any deviation from linearity in behavioural or neural data could not trivially be the consequence of how the score is defined.<\/p>\n To assess whether the subjects drew characters by reusing their own stroke primitives, we scored the fraction of character strokes that were high-quality matches to the subject\u2019s own primitives and the fraction that were high-quality matches to the other subject\u2019s primitives. If the fraction of matches to a subject\u2019s own primitives was high, and to the other subject\u2019s primitives was low, then this was evidence for recombining the subject\u2019s own primitives.<\/p>\n This assessment was done by assigning each stroke the label of its most similar primitive using the trajectory distance and then defining this as a high-quality match only if the trajectory distance was sufficiently low\u00a0(defined below).<\/p>\n First, each stroke was assigned its best-matching primitive, p<\/i>*, from a set of primitives (the choice of primitive set\u2014same or different subject\u2014depending on the analysis; see below):<\/p>\n $${p}^{\\ast }=\\mathop{\\text{argmin}}\\limits_{p}\\,d(s,{\\mu }_{p})$$<\/span><\/p>\n<\/div>\n where p<\/i> indexes the primitives, s<\/i> is the stroke trajectory, \u00b5<\/i>p<\/i><\/sub> is the mean trajectory for primitive p<\/i> (averaged over trials from the single-shape task), and d<\/i>(.,.) is the trajectory distance.<\/p>\n Second, the quality of the assignment of the stroke to p<\/i>* was scored as high if \\(d(s,{\\mu }_{{p}^{\\ast }}) < {{D}}_{max,{{\\text{p}}}^{\\ast }}\\)<\/span> and low if \\(d(s,{\\mu }_{{p}^{\\ast }})\\ge {{D}}_{max,{p}^{\\ast }}\\)<\/span>, where \\({{D}}_{max,{{\\text{p}}}^{\\ast }}\\)<\/span> is an upper\u00a0bound on trajectory distances that would be expected from trial-by-trial variation in primitive p<\/i>*. It is the 97.5th percentile of the distribution of trajectory distances from single-shape trials, determined separately for each primitive, which we consider as a good (arguably conservative) estimate of trial-by-trial variation. This is because the single-shape task presents no ambiguity as to what primitive needs to be drawn and can therefore be considered\u00a0\u2018ground\u00a0truth\u2019.<\/p>\n These steps assigned each stroke a class tuple (p<\/i>*,quality), where high quality was interpreted as the stroke matching the primitive set and low quality meant failure to match any primitive. Note that the non-uniformity of the frequency distribution of matches across primitives (Fig. 2o<\/a>) resembles non-uniformity seen for language and other behaviours94<\/a><\/sup>. Also, note that even the primitives with a low frequency of match (Fig. 2o<\/a>) can be considered legitimate matches because they satisfied the criteria to be labelled as high quality.<\/p>\n In summary analyses, we pooled all cases of high-quality matches into a single match class (regardless of the assigned primitive), and all low-quality matches into a single no-match class (Fig. 2p<\/a>). To test whether a given subject\u2019s character strokes aligned better with its own primitives compared with the other subject\u2019s primitives (Fig. 2p<\/a>), we performed the above analysis separately for all four combinations of stroke data (2 subjects)\u2009\u00d7\u2009primitives (2 subjects) using only images drawn by both the subjects.<\/p>\n The control analysis of testing primitive reuse using a simulated set of remixed primitives was performed as follows. We generated remixed primitive sets by mixing subparts of different primitives. Given two actual primitives, we extracted the first half of the first primitive (defined by the distance travelled) and the second half of the second primitive and connected them by aligning the offset of the first half to the onset of the second half, smoothing the connection with a sigmoidal weighting function. To sample a remixed primitive set, we first generated a pool of all possible remixed primitives using every possible pair of actual primitives. We then sampled (without replacement) a set of remixed primitives from this pool, keeping only remixed primitives that satisfied the following constraints: (1) no self-intersection, and (2) no excessively sharp turns, detected as curvature at any point along the inner 80% of the stroke exceeding 0.8. Curvature was defined in a standard manner as the inverse of the radius of curvature,<\/p>\n $$\\frac{\\dot{x}\\ddot{y}-\\dot{y}\\ddot{x}}{{({\\dot{x}}^{2}+{\\dot{y}}^{2})}^{\\frac{3}{2}}}$$<\/span><\/p>\n<\/div>\n where \\(\\dot{x}\\)<\/span> and \\(\\dot{y}\\)<\/span> are the velocity components and \\(\\ddot{x}\\)<\/span> and \\(\\ddot{y}\\)<\/span> are the acceleration components. Once a candidate set of remixed primitives was sampled, it had to pass further constraints: (1) each actual primitive contributed its first half or second half to a maximum of two remixed primitives; (2) the trajectory distance between no pair of remixed primitives was lower than the minimum trajectory distance between actual primitives; and (3) the trajectory distance between no pair of remixed primitive and actual primitive was less than the minimum trajectory distance between actual primitives. These constraints ensured that remixed primitives in each set were different from each other and from the actual primitives (visually apparent in Extended Data Fig. 5<\/a>).<\/p>\n We then labelled strokes from character drawings using each remixed primitive set using the same approach as for actual primitives, except the following. The analysis using actual primitives used trajectory distance thresholds (\\({{D}}_{max,{{\\text{p}}}^{\\ast }}\\)<\/span>)\u00a0determined empirically for each primitive based on single-shape trials (see above). Because remixed primitives were never drawn, a similar approach could not be used. Instead, thresholds for remixed primitives were assigned from the pool of thresholds for the actual primitives, in a manner meant to increase the stroke\u2013primitive match rate (and thus allow a stronger, or more conservative, test that the remixed primitives do not match the character strokes) by assigning the largest (most lenient) threshold to the worst-matching remixed\u00a0primitive, the second largest to the second worst, and so on, where worse means having larger average trajectory distance to character strokes.<\/p>\n To test whether each primitive was decodable from every other primitive based on single-trial kinematics (stroke trajectory) (Extended Data Fig. 2<\/a>), strokes were first converted from time series (x<\/i> and y<\/i> position) to eight-dimensional vectors as follows. Similar to the trajectory distance computation (above), strokes were normalized in time (linear interpolation to 50 time points) and space (rescaling each stroke, while maintaining its x<\/i>y<\/i> aspect ratio, to make the diagonal of its bounding box unit length\u00a01). The x<\/i> and y<\/i> coordinates were concatenated to a 100-dimensional vector and then reduced to 8 dimensions using principal component analysis (PCA). Decoding was performed on these 8D representations using a linear support vector machine classifier (SVC) (LinearSVC, scikit-learn (v.1.3.0), regularization parameter C set to 0.1), with 10-fold cross-validation. By performing this procedure for every pair of primitives (each time returning a single decoding accuracy score), we populated the matrix shown in Extended Data Fig. Behavioural task<\/h3>\n
Task overview<\/h4>\n
Early training<\/h4>\n
Trial structure<\/h4>\n
Scoring behavioural performance<\/h3>\n
Image similarity<\/h4>\n
Behavioural efficiency<\/h4>\n
Task-specific factors<\/h4>\n
Task types<\/h3>\n
Single-shape task<\/h4>\n
Multishape task<\/h4>\n
Character task<\/h4>\n
Behavioural data analysis<\/h3>\n
Preprocessing of touchscreen data<\/h4>\n
Computing the trajectory distance<\/h4>\n
Computing the image distance<\/h4>\n
Computing the primitive alignment score<\/h4>\n
Classifying strokes from the character task<\/h4>\n
Analysis of kinematic separability of primitives<\/h4>\n