{"title": "Toddler-Inspired Visual Object Learning", "book": "Advances in Neural Information Processing Systems", "page_first": 1201, "page_last": 1210, "abstract": "Real-world learning systems have practical limitations on the quality and quantity of the training datasets that they can collect and consider. How should a system go about choosing a subset of the possible training examples that still allows for learning accurate, generalizable models? To help address this question, we draw inspiration from a highly efficient practical learning system: the human child. Using head-mounted cameras, eye gaze trackers, and a model of foveated vision, we collected first-person (egocentric) images that represents a highly accurate approximation of the \"training data\" that toddlers' visual systems collect in everyday, naturalistic learning contexts. We used state-of-the-art computer vision learning models (convolutional neural networks) to help characterize the structure of these data, and found that child data produce significantly better object models than egocentric data experienced by adults in exactly the same environment. By using the CNNs as a modeling tool to investigate the properties of the child data that may enable this rapid learning, we found that child data exhibit a unique combination of quality and diversity, with not only many similar large, high-quality object views but also  a greater number and diversity of rare views. This novel methodology of analyzing the visual \"training data\" used by children may not only reveal insights to improve machine learning, but also may suggest new experimental tools to better understand infant learning in developmental psychology.", "full_text": "Toddler-Inspired Visual Object Learning\n\nSven Bambach1, David J. Crandall1, Linda B. Smith2, Chen Yu2\n\n1School of Informatics, Computing, and Engineering, 2Dept. of Psychological and Brain Sciences\n\nIndiana University Bloomington\n\n{sbambach, djcran, smith4, chenyu}@iu.edu\n\nAbstract\n\nReal-world learning systems have practical limitations on the quality and quantity\nof the training datasets that they can collect and consider. How should a system\ngo about choosing a subset of the possible training examples that still allows for\nlearning accurate, generalizable models? To help address this question, we draw\ninspiration from a highly ef\ufb01cient practical learning system: the human child.\nUsing head-mounted cameras, eye gaze trackers, and a model of foveated vision,\nwe collected \ufb01rst-person (egocentric) images that represent a highly accurate ap-\nproximation of the \"training data\" that toddlers\u2019 visual systems collect in everyday,\nnaturalistic learning contexts. We used state-of-the-art computer vision learning\nmodels (convolutional neural networks) to help characterize the structure of these\ndata, and found that child data produce signi\ufb01cantly better object models than\negocentric data experienced by adults in exactly the same environment. By using\nthe CNNs as a modeling tool to investigate the properties of the child data that may\nenable this rapid learning, we found that child data exhibit a unique combination of\nquality and diversity, with not only many similar large, high-quality object views\nbut also a greater number and diversity of rare views. This novel methodology of\nanalyzing the visual \"training data\" used by children may not only reveal insights\nto improve machine learning, but also may suggest new experimental tools to better\nunderstand infant learning in developmental psychology.\n\n1\n\nIntroduction\n\nAny learning system \u2014 human or machine \u2014 faces the challenge of building an accurate, general\nmodel of the world from a limited amount of training data. Both quality and quantity of the training\ndata are critical to successful learning. In the \ufb01eld of computer vision, for example, much of the\ndramatic recent progress in object recognition accuracy has been due to massive new datasets like\nImageNet [28]. While more training data is probably always better \u2014 very recent results show that\nrecognition models continue to improve even as datasets reach into the billions of images [19] \u2014\nboth human and machine learning systems in the real world are bounded by practical constraints on\nthe time and energy needed to collect and process a potentially in\ufb01nite amount of training data.\nThe usual approach in machine learning is to simply collect and use as much data as possible, with\nthe assumption that the training data is independently sampled from some underlying distribution\nthat is representative of the examples encountered in the real world. Since quality and quantity of\nthe training data are correlated, more quantity naturally leads to better overall quality. In the case of\nvisual recognition of object categories, for example, large datasets like ImageNet [28] contain not\nonly many instances of clear, canonical objects (e.g., side views of Toyota sedans) but also relatively\nfew instances of many less common objects (e.g., oblique views of Model T\u2019s). The larger the dataset\nis, the more likely it contains high-quality data points (e.g., uncommon instances) for generalization.\nIn this paper our goal is to delve into the data side of machine learning, but we draw inspiration\nfrom perhaps the best known visual learning system \u2014 the human child. We know that by the age of\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\ftwo, toddlers recognize instances of roughly 300 object categories [9]. They can also generalize a\nnewly learned label to instances that they have never seen before [16]. Even though computational\ndeep learning models trained with large databases of \u201cnatural\u201d images have matched and sometimes\noutperformed humans\u2019 remarkable visual abilities [13], human toddlers are still more ef\ufb01cient\nlearners: their visual learning relies on limited weakly-supervised training data from their individual\nexperiences, while deep learning models can use much larger quantities of supervised training data.\nRecent studies in developmental and cognitive psychology suggest that successful learning in toddlers\nlies in the quality of visual data they collect from their everyday activities [7]. In particular, while\nmost images in ImageNet and other computer vision datasets are photographs that were taken with\nconsumer cameras by adults, the visual information cast on a toddler\u2019s retina is from a \ufb01rst-person\nperspective. The visual properties of images from the \ufb01rst-person and third-person views are very\ndifferent [34]. Moreover, the visual experiences collected by toddlers are not a large collection of\nrandom pictures taken by many individuals in many locations and contexts, but are instead generated\nby a single person based on their everyday activities, and are thus much more coherent and correlated.\nWe believe that the unique visual properties and distributions of imagery perceived by human toddlers\nare key to their success in visual object learning.\nTo test this idea, we used head-mounted cameras and eye trackers to record an approximation\nof the visual stimuli that infants receive while playing with toys in a naturalistic, everyday play\nenvironment. We use these data to train state-of-the-art object models \u2014 deep convolutional neural\nnetworks (CNNs) \u2014 on an object recognition task, and study the performance of the networks as\nwe manipulate various properties of the training dataset. Our goal is not to build state-of-the-art\nobject classi\ufb01ers, nor is it to model the actual mechanism by which children learn; but to use CNNs\nto quantify and compare the information content of various datasets, i.e., to use them as data mining\nalgorithms to measure which properties of visual data lead to better visual object learning. By doing\nso, we believe the methodology we propose here in the long-term may not only reveal ways of\nimproving machine learning models by optimizing properties of training datasets, but also could lead\nto new computational tools to understand the underlying mechanisms behind infant learning.\n\n2 Related work\n\nA few recent studies have explored deep learning models in relation to data collected by human\nsubjects in experimental contexts [3, 12, 23, 27], and have already led to important \ufb01ndings in both\nthe machine and human learning \ufb01elds. On one hand, experimental psychology approaches provide a\nframework to understand hidden computational principles and properties of sophisticated algorithms\nsuch as deep neural networks [27, 30, 32]. On the other hand, deep learning models can be a useful\ntool to reveal the representations and computations required for human learners to solve hard learning\nproblems in the real world. Inspired by previous studies, the present study focuses on building\n\nparent view\n\neye camera\n\nscene camera\n\ntoddler view\n\neye camera\n\ny\n\nscene camera\n\nFigure 1: Our experimental setup. Child-parent dyads played together with a set of toys in a\nnaturalistic environment, while each wore head-mounted cameras to collect egocentric video and eye\ngaze positions (left). A stationary camera recorded from a third-person perspective (right).\n\n2\n\n\fconnections between visual object recognition in computer vision and toddler object learning in\ndevelopmental psychology.\nOur work connects with the data subset selection problem in machine learning, where the goal\nis to \ufb01nd a good subset of training examples [10, 11, 17, 21, 33, 37], often in the context of active\nlearning [14,20]. Most of this work formulates the problem mathematically for speci\ufb01c classi\ufb01ers with\nspeci\ufb01c objective functions. Our work is more generally related to the rich literature in data mining\nand machine learning on choosing subsets of data having certain properties. Of course, the most\ncommon approach to reducing the size of a dataset while preserving its statistical properties is random\nsampling. Other approaches include algorithms that represent a dataset with a few representative\npoints [15, 24], try to correct for biases in training data [6], minimize or maximize the variation of\ndata [4,8,25], sample from inliers while ignoring outliers [5], and so on. Instead of speci\ufb01c algorithms\nor mathematical formulations, we are interested in identifying more fundamental principles of what\nmakes a good training dataset for visual object recognition and how children may naturally generate\nthem in everyday contexts.\n\n3 Methods\n\n3.1 Data and Data collection\n\nTo closely approximate the training data used in toddler object learning, we collected visual data\nfrom everyday toy play \u2014 a context in which toddlers naturally learn about objects and their names.\nWe used an experimental setup in which we placed a camera on a toddler\u2019s head to collect egocentric\nvideo of their \ufb01eld of view as they played with toy objects with their parent, as shown in Figure 1.\nWe also used a head-mounted eye tracker to record their visual attention. These devices allowed\nus to record both egocentric video from the toddler\u2019s perspective as well as their gaze point on\na moment-by-moment basis. We also collected video and gaze data from the parent in the same\ninteraction. This experimental setup has been used successfully in studying infant perception and\naction [36], language learning [34], and social interaction [35]. Our study was reviewed and approved\nby the IRB at our institution.\nTwenty-six child-parent dyads participated in our study. Each dyad was brought into a room with a\nset of 24 toys scattered on the \ufb02oor; we used the same toys as in [3]. The children and parents were\ntold to play with the toys, but no more speci\ufb01c directions were given. The children ranged in age from\n15.2 to 24.2 months (\u00b5=19.4 months, =2.2 months). We collected \ufb01ve synchronized videos per dyad\n(head camera and eye camera for child, head camera and eye camera for parent, and a third person\nview camera \u2013 see Figure 1). The \ufb01nal dataset contains roughly 200 minutes of video, with each dyad\ncontributing different amounts of data ranging from 3.4 minutes to 11.6 minutes (\u00b5=7.5 minutes,\n=2.3 minutes). The head-mounted cameras recorded video at 30 frames per second and 480 \u21e5 640\npixels per frame, with a horizontal \ufb01eld of view of about 70 degrees, and we used eye trackers from\nPositive Science [1]. We followed validated best practices for mounting the head cameras so as to\nbest approximate subjects\u2019 actual \ufb01rst-person views, and to calibrate the eye trackers. The collected\ndata were used to build different toy object training sets, as detailed in Sections 3.2 to 3.5.\nWe also used a separate image dataset (provided by [3]) of the same 24 toy objects captured by a\nthird person camera in a controlled environment, some examples of which are shown in Figure 2c.\nThe dataset consists of each object systematically photographed up-close against a black background\nfrom 128 viewpoints (different angles and distances), for a total of 3,072 images. These images are\nused as the test set for the experiments in Section 4.\n\n3.2 Detecting Object Looks\n\nFrom continuous gaze data, we manually coded, for each frame and each subject (toddlers and\nparents), which object the subject was attending to (if any). We de\ufb01ned an object look as either a\ntoddler or parent continuously attending to the same object before switching their attention to another\nobject. In total, there were 4,553 object looks for the toddlers and 5,052 object looks for the parents.\nThe average object look duration was 1.86 seconds (56 frames) for toddlers and 1.40 seconds (42\nframes) for parents. Some objects were attended more than others: the most popular toy attracted\nabout four times as many looks as the least popular of the 24 toys. In this study, we used only the\nframes in which participants were looking at one of the 24 toys (e.g., we excluded looks at the other\n\n3\n\n\fperson\u2019s face), totaling 258,250 frames for children and 217,290 frames for parents. The training data\nin our simulations (described below) consisted of the object instances attended to in these frames.\n\n3.3 Detecting Objects\n\nTo avoid labor-intensive manual annotation, we used an automatic process to identify positions and\nsizes of objects visible in each frame. In particular, we used YOLO [26], a well-established detector\nthat offers a good compromise between computational cost and accuracy. To train the model on\nour 24 objects, we manually annotated a small subset of about 1,200 frames (sampled from 15\nrandomly-chosen subjects) with bounding boxes. Although our models and YOLO detections are not\nperfect, the results are generally very good; for 97% of our (manually-annotated) child looks, YOLO\ncorrectly identi\ufb01ed the attended object in at least one frame, while 83% of the parent looks had a\ncorrect YOLO recognition for at least one frame. The lower accuracy for parents was presumably\nbecause objects tend to be smaller in the parent\u2019s \ufb01eld of view (discussed below).\n\n3.4 Simulating Acuity\n\nEgocentric video captured by head-mounted cameras provides a good approximation of the \ufb01eld of\nview of the wearer. However, the human visual system exhibits well-de\ufb01ned contrast sensitivity due\nretinal eccentricity [22]: the area centered around the gaze point (the fovea) captures a high-resolution\nimage, while the imagery in the periphery is captured at dramatically lower resolution due to its lesser\nsensitivity to higher spatial frequencies. As a result, the human visual system does not process all\npixels equally in a \ufb01rst-person view image, but instead focuses more on the pixels around the fovea.\nTo closely approximate the visual signals that are \u201cinput\u201d to a toddler\u2019s learning system, we simulated\nfoveated vision in the egocentric view by applying a blurring function that preserves \ufb01ne details\nonly at the center of gaze. To do this, we started with the set of egocentric camera frames described\nabove, where each frame is annotated with the xy-coordinate of the center of gaze as well as the class\nlabel of the attended object. We applied the method of Perry and Geisler [22] to simulate the effect\nof foveated visual acuity on each frame individually using their software implementation [2]. The\nbasic idea is to preserve the original high-resolution image at the center of gaze, and increase blur\nprogressively towards the periphery, as shown in Figures 2a and 2b. This technique applies a model\nof what is known about human visual acuity and has been validated with human psychophysical\nstudies [22].\nAfter gaze-based blurring to simulate acuity, we cropped out the area centered around the gaze\ncoordinate to produce a training sample. We cropped out patches of different sizes, in order to\nsimulate different amounts of visual data that may be processed by the toddler\u2019s learning system. In\nparticular, we started with crops of 30 width (roughly corresponding to the area with high acuity)\nand increased the size in 10 increments up to 70 (the \ufb01eld of view of the head camera); examples\nare shown in Figures 2a and 2b. This procedure yields a total of 10 different datasets, corresponding\nto \ufb01ve different crop sizes (30, ..., 70) for each of the two subject types (toddlers and parents).\n\n3.5 Convolutional Neural Network models\n\nIn our experiments the goal is not to achieve the best possible accuracy using the latest recognition\nalgorithm: we are interested in using machine learning classi\ufb01ers to understand and characterize the\nproperties of different training datasets, not to produce classi\ufb01ers to be actually used for recognition.\nOur goal is also not to model the actual mechanism by which children learn: we are interested in\ncharacterizing the information and structure embedded in the data that could be learned by a good\nalgorithm, human or computational. In other words, we use machine learning as a data mining tool\nthat is useful for quantifying properties of visual data.\nWe use a well-known, state-of-the-art deep learning model, a convolutional neural network with the\nVGG16 [29] architecture, for image classi\ufb01cation. Because our goal is not to optimize performance\nof the classi\ufb01er itself, we use exactly the structure proposed in [29], except that we change the output\nlayer to have 24 classes. We pre-trained the network on ImageNet [28], and used those weights to\n\ufb01ne-tune the network on our training data, back-propagating across the whole network. Because the\ndistribution of training examples (attended objects) is highly non-uniform, we used a categorical\ncross-entropy loss function that weighted the loss for each class to be inversely proportional to the\n\n4\n\n\fscene camera, gaze, a\u01a9ended object\n\nsimulated acuity\n\nscene camera, gaze, a\u01a9ended object\n\nsimulated acuity\n\n70\u00b0 \n\n60\u00b0 \n\n50\u00b0 \n\n40\u00b0 \n\n30\u00b0 \n\n70\u00b0 \n\n60\u00b0 \n\n50\u00b0 \n\n40\u00b0 \n\n30\u00b0 \n\n(a) Toddler data\n\n(b) Parent data\n\n(c) Clean third-person data\n\nFigure 2: We use head-mounted cameras to collect egocentric video and eye gaze from parents and\nchildren. Moments of sustained attention on objects are annotated by human coders. (a,b): We\nsimulate foveated vision using the eye gaze position, and generate training examples with varying\namounts of context by cropping at different scales around the gaze point. (c): Clean images of all 24\ntoy objects in the experiment, randomly chosen from the test dataset.\n\nnumber of training examples of that class,\n\nL(\u2713) = \n\n1\nNj\n\n(yi = j) ln fj(xi,\u2713 ),\n\nNXi=1\n\n24Xj=1\n\nwhere N is the total number of training examples, Nj is the number of training examples for class\nj, yi is the correct class label for the i-th example, and fj(xi,\u2713 ) is the value of the output neuron\ncorresponding to class j when the network with weights \u2713 is applied to example xi. We used a\nstandard stochastic gradient descent optimizer with a learning rate of 0.001, momentum of 0.9, and\na batch size of 64 images. All training images were resized to 224 \u21e5 224 pixels, and we did not\nperform any data augmentation (e.g., left-right re\ufb02ections or random croppings) since we wanted to\nuse just the data that the infant learners receive.\nOur goal is to quantify the information contained in a training dataset \u2014 i.e., how well an agent\ncould learn to generalize given the training data \u2014 so we trained on the images derived from the\nhead camera videos, but validated and tested our models on the \u201cclean\u201d object dataset described\nin Section 3.1 and Figure 2c. The motivation behind this is to train each \ufb01rst-person dataset to the\npoint where it best generalizes to viewpoint-independent objects instances. We thus test each network\non the clean dataset after every epoch and stop training once the accuracy has not increased for at\nleast two epochs, and report the highest overall classi\ufb01cation accuracy achieved up to that point.\nBecause training is stochastic, we independently trained 10 networks for every condition and report\nthe average classi\ufb01cation accuracy as well as 95% con\ufb01dence intervals.\n\n4 Experiments and Results\n\n4.1 Quality of Toddler Data\n\nWe \ufb01rst compared the visual data perceived by toddlers with that perceived by their parents during\nthe same interactions in the same environment. Figure 3a summarizes different views of one attended\nobject (a red helmet), indicating a clear difference in object size and variation between the two groups.\nIn light of this observation, we \ufb01rst quanti\ufb01ed and compared two visual properties of object instances\nin the toddler\u2019s view with those in the parent\u2019s view.\nFirst, as shown in Figure 3a, the distribution of object size was strikingly different across the two\nviews: most instances of attended objects in the parent\u2019s view are smaller than 10% of the \ufb01eld of\nview, but objects in the toddler\u2019s view are much larger, with a large proportion of instances greater\nthan 20% of the \ufb01eld of view. This may be due to several causes: children are shorter and closer to\nthe ground, have shorter arms that make held objects appear larger, and may bring objects closer to\n\n5\n\n\f(a) Histograms of object size\n\n(b) Histograms of object variability\n\nFigure 3: Statistics of data collected by toddler and parent head cameras, with ImageNet as a baseline.\n(a): Distribution of training examples by object size (measured as the fraction of \ufb01eld of view). (b):\nDistribution of variation in the training examples (measured by distance between GIST features of\npairs of examples of the same object/class).\n\ntheir face than parents. But the effect is to create large objects in view (similar to ImageNet), which\nin turn create high-resolution images for learning \u2014 an idea we will test in Section 4.2.\nSecond, we compared visual similarity of object instances in order to quantify the diversity of the\ntraining data. In particular, we extracted GIST features [31] from each object instance and computed\npairwise GIST distances (L2 norm) across all instances within each object category. We chose GIST\nbecause it is a low-level feature (as opposed to more semantic deep features) that is sensitive to the\nspatial orientation of an object; we wanted a distance metric such that two instances of the same\nobject viewed from similar angles would have a small distance, while different views of the same\nobject would have a large distance. As shown in Figure 3b, pairwise visual distances in parent data\nare rather small, indicating that object instances are similar to each other. For toddler data, there is\nalso a portion of similar instances, but a bigger tail of instances that are not so similar to each other,\nshowing variability within the training set. Thus, the combination of clustering and variability creates\na unique distribution in toddler data which may bene\ufb01t visual object learning \u2014 a conjecture that will\nbe tested in Section 4.3. Note that the variability between objects in our data is expected to be much\nsmaller than in ImageNet as we are comparing instances of the same physical object, not instances of\ndifferent objects of the same object category.\nGiven the different visual properties observed in the views from toddlers and parents, we next\nexamined whether the special properties of the toddler data can be used to create better object\nrecognition models. To do this, we trained a set of VGG16 networks on only the child data, and\nindependently trained another set on only the parent data, following the methodology and parameters\ndetailed in Section 3.5. Figure 4b summarizes the object recognition accuracy of these models on the\nclean test set, showing that the toddler models indeed perform much better than the parent-trained\nmodels across all training conditions (i.e. using different-sized crops around the gaze center as input).\nWe also investigated the effect of the simulated acuity by repeating all experiments without blurring\nimages. Our results show that blurring was bene\ufb01cial for learning only when objects were small and\nviews were cluttered (e.g., for adults at 60/70), suggesting that foveated vision can help learning\nby zooming into the area of focus in cluttered views, but can also hinder learning by lowering the\noverall resolution of the target object. In both cases, either with or without blurring, the main result\n(that toddler data lead to better model performance) remains true. We note here that similar \ufb01ndings\nhave been reported previously in [3]. However, the previous study was based on a smaller number of\nparticipants and did not include eye tracking, but instead trained models with all visible objects in the\n\ufb01rst-person scene, a less realistic approach. In contrast, the present study used only visually attended\nobjects with moment-by-moment acuity simulations, which is a much closer approximation of the\nvisual data collected by toddler learners and thus allows us to look deeper into what properties of\ntoddler data lead to better object recognition.\n\n4.2 Quality in Object Size\n\nThe results in the last section suggest that there are special properties of toddler training data that\ncan lead to better visual learning. Using only toddler data we now investigate these properties more\nclosely, starting with the size of attended objects relative to the \ufb01eld of view. To test the effect of size,\nwe \ufb01rst randomly sampled 10,000 frames for which we have object bounding boxes (as detected by\nYOLO) for the attended object. We approximated the object size with the size of the bounding box.\nWe created two training sets of 5,000 frames each, such that one contained objects smaller than the\nmedian and the other contained the larger objects. The median object size was about 10% of the \ufb01eld\n\n6\n\n\ftoddler\u2019s view\n\nadult\u2019s view\n\n(a) Object instances (shown to scale, starting with largest)\n\n(b) Object recognition accuracy\n\nFigure 4: Comparison of egocentric training data collected by toddlers and parents. (a): Image crops\nof one example object captured by toddlers and parents are visually very different, in terms of the\nobject size and diversity of views. (b): CNNs trained on the data collected by toddlers signi\ufb01cantly\noutperform those trained with parent data, when tested on a third independent test set.\n\nbig objects\n\nsmall objects\n\n(a) Examples of big and small attended objects (70 crops)\n\n(b) Object recognition accuracy\n\nFigure 5: Effect of object size on the quality of the training data. (a): We split the child training\ndataset into small and large instances. (b): The subset consisting of large instances led to an object\nmodel that performed signi\ufb01cantly better on our test dataset.\n\nof view, and the instances in the two subsets had mean sizes of 6.8% and 19.6% of the \ufb01eld of view,\nrespectively. Note that the YOLO boxes were used only to split the instances into subsets, and the\ntraining samples themselves were still based on \ufb01xed-size crops around the center of gaze. Figure 5a\nshows examples (70 crops) of training instances with small and large attended objects.\nWe then trained separate but identical VGG16 networks on the small and large objects collected from\nthe child head cameras. As shown in Figure 5b, the model trained with the large objects achieved\nsigni\ufb01cantly better accuracy on the test dataset than that trained with the small objects. Given that the\nobject instances used in training were all resized to 224 \u21e5 224 pixels before being fed into the CNNs,\nthe results here were not due to a direct size effect, but presumably because larger objects created\nhigher-\ufb01delity instances after being resized, and this better image quality led to better learning.\n\n4.3 Quality in object variability\n\nAs shown in Section 4.1, another visual property in toddler data that may contribute to successful\nobject learning is the distribution of object views. Clustering in training data could be a useful\nproperty, as it would allow a learning system to detect a central prototype of an object category.\nVariability among examples could also help by encouraging generalization. As shown in Figure 3b,\nthe distribution of toddler data seems to contain both properties, with a large number of highly similar\ninstances but also a tail of diverse ones. Our working hypothesis is that the speci\ufb01c combination of\nproperties encoded in toddler data may be key for children\u2019s successful object learning.\nTo test this idea, we created three training subsets with the same amount of training data in each: (1)\na set of similar objects containing object instances that shared similar appearance; (2) a diverse set\ncontaining object instances that were not similar but had different appearances; and (3) a random\n\n7\n\n\fs\nt\nc\ne\nj\nb\no\n \ne\ns\nr\ne\nv\ni\nd\n\ns\nt\nc\ne\nj\nb\no\n \nr\na\nl\ni\n\nm\ni\ns\n\n(a) Object instances in GIST space\nFigure 6: We study the effect of visual diversity by creating two subsets from the toddler data, one\nhomogeneous and one diverse. (a) Visualization of the similar and diverse object subsets for one\nobject in GIST space. (b) Sample instances from the both subsets. (c) The diverse subset led to a\nbetter model than the similar one, but the original dataset having a mixture of the two performed best.\n\n(c) Object recognition accuracy\n\n(b) Object examples\n\nsubset of the original set that represents the true object-view distribution generated by toddlers. In\nparticular, we \ufb01rst randomly sampled 10,000 frames for which we had YOLO boxes for the attended\nobject. We cropped out each object using its box and then computed its GIST [31] feature. Using\nthe similarity measure based on GIST, we created a \u201csimilar\u201d set consisting of the 25% of instances\nwith (approxiately) the minimum total pairwise distance. We also created a \u201cdiverse\u201d set containing\nthe 25% of all instances that (approximately) maximized the total distance between all pairs of\ninstances. Finding these subsets involves solving hard computational problems (e.g., the max-sum\ndiversi\ufb01cation problem [4]). We use a greedy approximation in which we start with the pair of\ninstances having the largest (or smallest) distance, and then iteratively add the next instance with the\nlargest (or smallest) distance from the centroid of the points selected so far. Figure 6 shows sample\nimages from both sets, as well as a visualization of their distribution in GIST space (projected into\ntwo dimensions using t-SNE [18]).\nTraining the same model using these three subsets, we found that the diverse set outperformed the\nsimilar set on our test set, as shown in Figure 6c. This result suggests that seeing different examples\nhelps the model to generalize. More interestingly, the original set outperformed both the similar and\ndiverse sets. Our results suggest that the data created by toddlers, which consists of a mix of both\nsimilar and dissimilar instances, is a unique combination of clustering and variability that may be\noptimal for object recognition.\n\n5 Conclusion\n\nDeep learning models have made remarkable progress in matching humans\u2019 visual abilities. However,\nthose models rely on large quantities of supervised training data to achieve superior performance.\nInspired by toddler learning, the present paper focused on quality of data to understand which funda-\nmental properties in visual training data can lead to successful learning in visual object recognition \u2014\none of the most challenging tasks in computer vision and machine learning. Towards this goal, we\nhave conducted a series of simulations which systematically examined how different properties of\ntraining data lead to different learning outcomes. We found that image data from toddlers\u2019 egocentric\nviews contains unique properties and distributions that are critical for successful learning. This is\nthe \ufb01rst study that has applied deep learning models as formal models to understand visual object\nrecognition in young children, and we believe this methodology can be informative not only for\nstudies of machine learning but also of human learning. Our \ufb01ndings suggest that in everyday toy play,\ntoddlers create their own data with useful properties for learning. While their internal information\nprocessing capabilities may not be as sophisticated as those of adults, they have high-quality training\ndata to solve hard learning problems.\nMore generally, our work suggests that deep learning models may not have to rely on large quantities\nof training data to reach good performance, but that a smaller number of carefully-selected, high-\nquality examples may be suf\ufb01cient. Critically, developing a systematic way to link properties of\ntraining data with the learning mechanisms used to process data could eventually allow us to \ufb01nd\nmore ef\ufb01cient ways to train machine learning models.\n\n8\n\n\fAcknowledgments\n\nThis work was supported by the National Science Foundation (CAREER IIS-1253549) and the\nNational Institutes of Health (R01 HD074601, R01 HD093792), as well as the IU Of\ufb01ce of the Vice\nProvost for Research, the College of Arts and Sciences, and the School of Informatics, Computing,\nand Engineering through the Emerging Areas of Research Project \u201cLearning: Brains, Machines,\nand Children.\u201d We would like to thank Drew Abney, Esther Chen, Steven Elmlinger, Seth Foster,\nLauren Slone, Catalina Suarez, Charlene Tay, and Yayun Zhang for helping with the collection of the\n\ufb01rst-person toy play dataset.\n\nReferences\n[1] http://www.positivescience.com.\n[2] http://www.svi.cps.utexas.edu/software.shtml.\n[3] Sven Bambach, David Crandall, Linda Smith, and Chen Yu. Active viewing in toddlers facilitates visual\nobject learning: An egocentric vision approach. In Annual Conference of the Cognitive Science Society\n(CogSci), 2016.\n\n[4] Alfonso Cevallos, Friedrich Eisenbrand, and Rico Zenklusen. Local search for max-sum diversi\ufb01cation. In\n\nProceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, 2017.\n\n[5] Moses Charikar, Sudipto Guha, \u00c9va Tardos, and David B. Shmoys. A constant-factor approximation\nalgorithm for the k-median problem. In Proceedings of the Thirty-\ufb01rst Annual ACM Symposium on Theory\nof Computing, 1999.\n\n[6] Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. Smote: Synthetic\n\nminority over-sampling technique. J. Artif. Int. Res., 16(1):321\u2013357, June 2002.\n\n[7] Elizabeth M. Clerkin, Elizabeth Hart, James M. Rehg, Chen Yu, and Linda B. Smith. Real-world visual\nstatistics and infants\u2019 \ufb01rst-learned object names. Philosophical Transactions of the Royal Society of London\nB: Biological Sciences, 372(1711), 2017.\n\n[8] Ehsan Elhamifar, Guillermo Sapiro, and S. Shankar Sastry. Dissimilarity-based sparse subset selection.\n\nIEEE Trans. Pattern Anal. Mach. Intell., 38(11), November 2016.\n\n[9] Michael C. Frank, Mika Braginsky, Daniel Yurovsky, and Virginia A Marchamn. Wordbank: an open\n\nrepository for developmental vocabulary data. Journal of Child Language, 44:677\u2013694, 2017.\n\n[10] Y. Fu, X. Zhu, and A. K. Elmagarmid. Active learning with optimal instance subset selection. IEEE\n\nTransactions on Cybernetics, 43(2):464\u2013475, 2013.\n\n[11] Chris Gathercole and Peter Ross. Dynamic training subset selection for supervised learning in genetic\nprogramming. In Proceedings of the International Conference on Evolutionary Computation. The Third\nConference on Parallel Problem Solving from Nature: Parallel Problem Solving from Nature, 1994.\n\n[12] Michelle R Greene, Christopher Baldassano, Andre Esteva, Diane M Beck, and Li Fei-Fei. Visual scenes\n\nare categorized by function. Journal of Experimental Psychology: General, 145(1):82, 2016.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into recti\ufb01ers: Surpassing\nhuman-level performance on imagenet classi\ufb01cation. In IEEE International Conference on Computer\nVision (ICCV), pages 1026\u20131034, 2015.\n\n[14] S. J. Huang, R. Jin, and Z. H. Zhou. Active learning by querying informative and representative examples.\n\nIEEE Transactions on Pattern Analysis and Machine Intelligence, 36(10):1936\u20131949, Oct 2014.\n\n[15] A. K. Jain, M. N. Murty, and P. J. Flynn. Data clustering: A review. ACM Comput. Surv., 31(3):264\u2013323,\n\nSeptember 1999.\n\n[16] Barbara Landau, Linda B Smith, and Susan S Jones. The importance of shape in early lexical learning.\n\nCognitive development, 3(3):299\u2013321, 1988.\n\n[17] Hui Lin and Jeff Bilmes. How to select a good training-data subset for transcription: Submodular\nactive selection for sequences. In Tenth Annual Conference of the International Speech Communication\nAssociation, 2009.\n\n[18] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning\n\nResearch, 9(Nov):2579\u20132605, 2008.\n\n[19] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ash-\nwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining.\narXiv:1805.00932, 2018.\n\n9\n\n\f[20] Hieu T. Nguyen and Arnold Smeulders. Active learning using pre-clustering. In Proceedings of the\n\nTwenty-\ufb01rst International Conference on Machine Learning, 2004.\n\n[21] Robert M Nosofsky, Craig A Sanders, Xiaojin Zhu, and Mark A McDaniel. Model-guided search for\noptimal natural-science-category training exemplars: A work in progress. Psychonomic bulletin & review,\npages 1\u201329, 2018.\n\n[22] Jeffrey S. Perry and Wilson S. Geisler. Gaze-contingent real-time simulation of arbitrary visual \ufb01elds. In\n\nHuman vision and electronic imaging VII, 2002.\n\n[23] Joshua C Peterson, Joshua T Abbott, and Thomas L Grif\ufb01ths. Adapting deep network features to capture\n\npsychological representations. arXiv preprint arXiv:1608.02164, 2016.\n\n[24] Mark Plutowski, Garrison Cottrell, and Halbert White. Learning mackey-glass from 25 examples, plus or\n\nminus 2. In Advances in Neural Information Processing Systems (NIPS), pages 1135\u20131142, 1994.\n\n[25] Adarsh Prasad, Stefanie Jegelka, and Dhruv Batra. Submodular meets structured: Finding diverse subsets\n\nin exponentially-large structured item sets. arXiv:1411/1752, 2014.\n\n[26] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You Only Look Once:\nUni\ufb01ed, Real-Time Object Detection. In IEEE Conference on Computer Vision and Pattern Recognition\n(CVPR), 2016.\n\n[27] Samuel Ritter, David G. T. Barrett, Adam Santoro, and Matt M. Botvinick. Cognitive psychology for deep\n\nneural networks: A shape bias case study. arXiv:1706.08606, 2017.\n\n[28] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang,\nAndrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge.\nInternational Journal of Computer Vision (IJCV), 115(3):211\u2013252, 2015.\n\n[29] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition.\n\nCoRR, abs/1409.1556, 2014.\n\n[30] Linda B. Smith and Lauren K. Slone. A developmental approach to machine learning? Frontiers in\n\nPsychology, 8:2124, 2017.\n\n[31] Antonio Torralba. Contextual priming for object detection. International Journal of Computer Vision\n\n(IJCV), 2003.\n\n[32] Panqu Wang and Garrison Cottrell. A computational model of the development of hemispheric asymmetry\n\nof face processing. In Annual Conference of the Cognitive Science Society (CogSci), volume 35, 2013.\n\n[33] Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In\n\nProceedings of the 32nd International Conference on Machine Learning, pages 1954\u20131963, 2015.\n\n[34] Chen Yu and Linda B. Smith. Embodied attention and word learning by toddlers. Cognition, 125(2):224\u2013\n\n262, 2012.\n\n[35] Chen Yu and Linda B Smith. The social origins of sustained attention in one-year-old human infants.\n\nCurrent Biology, 26(9):1235\u20131240, 2016.\n\n[36] Chen Yu and Linda B Smith. Hand\u2013eye coordination predicts joint attention. Child development,\n\n88(6):2060\u20132078, 2017.\n\n[37] Xiaojin Zhu. Machine teaching: An inverse problem to machine learning and an approach toward optimal\n\neducation. In AAAI, pages 4083\u20134087, 2015.\n\n10\n\n\f", "award": [], "sourceid": 624, "authors": [{"given_name": "Sven", "family_name": "Bambach", "institution": "The Research Institute at Nationwide Children's Hospital"}, {"given_name": "David", "family_name": "Crandall", "institution": "Indiana University"}, {"given_name": "Linda", "family_name": "Smith", "institution": "Indiana University"}, {"given_name": "Chen", "family_name": "Yu", "institution": "Indiana University"}]}