Human hand kinematic data during feeding and cooking tasks

This work presents a database of human hand kinematics containing data collected during the performance of a wide variety of activities of daily living involving feeding and cooking. The data were recorded using CyberGlove instrumented gloves on both hands measuring 18 degrees of freedom on each. A total of 20 subjects participated in each part of the experiment, and the objects and their arrangement were the same across subjects, although they performed the tasks in a natural non-directed way. This dataset contains a total of 1160 continuous calibrated recordings taken at 100 Hz during the performance of the tasks, with filtered signal. Statistical descriptive analyses from these data are presented. This database can be useful for machine learning purposes and prostheses control, as well as for the characterization of healthy human hand kinematics.

www.nature.com/scientificdata www.nature.com/scientificdata/ Table 1 shows an overview of different datasets focused on hand kinematics and their characteristics. In this paper we present the KINE-ADL BE-UJI Dataset 21 , which contains a total of 1160 recordings with anatomical angles of both hands while performing feeding and cooking activities using a large variety of products. Experiments were performed by 20 healthy subjects while wearing CyberGlove instrumented gloves on both hands, 18 DOF being recorded in each hand at a frequency of 100 Hz. The main contribution of this dataset compared to others is the variety of objects used (66 objects), the in-depth study of representative feeding and cooking tasks (58 tasks, divided into 178 actions) and the freedom given to the subjects to perform the tasks. Moreover, the data were collected from both hands, which allows the study of hand coordination. It is also important that the sample of subjects was selected so as to be representative of the healthy adult population (with a controlled proportion of laterality and gender). Furthermore, the data presented is standardized, as it is presented as anatomical angles following the ISB sign criteria 22 . The dataset consists of a Matlab/GNU Octave data structure (.mat) (provided also in .csv format) with kinematic data and data about the subjects recruited (age, gender, laterality, weight, height, hand length, hand width and active range of motion (AROM) measured for each DoF). This .mat file is accompanied by a guide where information regarding the environment, tasks, objects, data acquisition system and file structure is detailed, thereby allowing the classification of information regarding these parameters.

Methods
Study participants. The study consisted of two experiments (A and B), with 20 subjects (10 males, 10 females) participating in each experiment. Only 15 subjects participated in both experiments, so that the total amount of subjects recruited was 25. In both experiments, two of the 20 subjects were left-handed. The mean age of subjects recruited was 35.5 ± 7.67 years in experiment A and 38.05 ± 9.52 years in experiment B. The criteria used to select subjects were gender parity in overall data, age between 20 and 65, no reported upper limb pathologies and laterality representative of the overall population (20% of data from left-handed individuals). Before the experiments, all participants gave their written informed consent. All the experiments were performed in accordance with the Ethics Committee of the Universitat Jaume I. acquisition setup. Instrumentation. Data acquisition was performed using two CyberGlove (CyberGlove Systems LLC) instrumented gloves (CyberGlove II on the right hand and CyberGlove III on the left hand) connected to a laptop. Each of these gloves has 18 strain gauges that allow the anatomical angles of the underlying joints to be determined. The angle rotated by each joint with respect to the reference posture (hands resting flat on a table, with the fingers and thumb close together, and the middle fingers aligned with the forearms) is then calculated from these signals, according to a previously validated calibration protocol 23 . Furthermore, all the experiments were recorded on video, so as to be able to check the performance of the task when subsequently required.
Environment. The tasks were performed in a laboratory, within an environment that simulated a kitchen ( Fig. 1), composed of: a refrigerator (Scenario 1), a high cabinet (Scenario 2), shelves (Scenario 3), a small worktop (Scenario 4), a sink and a rubbish bin (Scenario 5), a large worktop (Scenario 6), a low cabinet with a drawer in its upper part and shelves in the lower part, which has a door (Scenario 7), a table and a chair (Scenario 8) and an oven (Scenario 9).
Objects. A total of 66 objects were used to perform the tasks in the experiments (further information regarding their characteristics can be found in the guide attached to the dataset). The objects were chosen so as to be representative of those most commonly used in cooking and feeding tasks, and were checked to ensure they covered the cooking and feeding objects from the Yale-CMU-Berkeley Object and Model Set 24 , proposed by Calli et al. Some of the objects used were not real, in order to prevent the gloves from getting stained or wet. For example, the eggs to be broken had been previously emptied through a small hole made in the shell. All liquids were replaced by water, and materials such as flour or sugar that could have stained the gloves were replaced by durum wheat semolina. Pieces of polystyrene or cardboard were used to simulate biscuits, bread or crisps. The initial location of the objects in each scenario can be found in the detailed guide attached to the database. Figure 2 shows an overview of the objects used. www.nature.com/scientificdata www.nature.com/scientificdata/ acquisition protocol. The main dimensions of the hands were measured before helping the subject to put on the instrumented gloves following the manufacturer's instructions. Participants were given clear instructions about how to perform the task, and they were told to start and end each task in the same posture: hands lying relaxed at both sides of the body for tasks performed in a standing posture, and hands lying relaxed on the table when sitting. While carrying out each task, the operator marked (or labelled) the time stamp of some specific events (using the glove software) that were later used to separate different phases or actions.
Recorded tasks. Two experiments (A and B) were performed. In experiment A, the activities performed were: preparing and having breakfast, baking a cake and cooking omelets. In experiment B, the activities were: setting the table, clearing the table and washing the dishes, making coffee and preparing a simple meal, considering the whole process of performing each task (taking the products from the different scenarios, transporting them, opening/using them and, in some cases, putting them back in their place). Furthermore, all these tasks were separated into different recordings (e.g. using the toaster or pouring and drinking milk), and these recordings were also separated into different elementary tasks (e.g. object grasping, manipulation such as opening tins/jars/ bottles, transportation of objects, pouring liquid/solid substances, eating/drinking and other relevant actions). Therefore, experiment A was divided into 33 recordings and experiment B consisted of 25 (a description of all the recordings can be seen in Tables 2 and 3). Further information regarding the elementary tasks considered in each recording can be found in the guide attached to the dataset.
Some of the recordings were performed with the subject standing and others while sitting on a chair (as specified in Tables 2 and 3). Only the eating or drinking activities were simulated, by just bringing the food close to the mouth, and this has been indicated in the task description. The rest of the tasks were performed with realistic objects, and subjects were free to perform the tasks in the way they preferred.
Elementary tasks. As mentioned previously, each of the recordings (R) (33 recordings in experiment A and 25 in B) is composed of different elementary tasks. For example, in the activity of having breakfast (consisting of 11 records, as seen in Table 2) record R = 106 (pouring and drinking milk) is composed of four elementary tasks: opening the carton, pouring, closing the carton and drinking (see Table 4). For an unambiguous identification of each of the tasks, a unique ID was assigned to each elementary task, with a total of 99 elementary tasks in experiment A and 79 in B (178 elementary tasks altogether). All the elementary tasks involved grasping or manipulating a product or element with the hands, except for some cases where the subject moved without handling anything, which were labelled as "Displacement without manipulation". For each elementary task, the record considers all time instants since the object was grasped until it was released. In those cases in which the object was released in a specific place or transported to a specific location in the scenario, this location is specified in the description of the elementary task. In all other cases, the release was performed on the surface closest to the subject (table, worktop, etc.). www.nature.com/scientificdata www.nature.com/scientificdata/ Active range of motion (AROM). After performing all the experiments, subjects were asked to perform a set of postures 25 in order to measure their AROM of the joints of both hands, which are presented in the .mat file, where subject information is also provided.
Signal processing. Angles calculation. Joint angles were calculated from raw data collected according to the calibration protocol proposed in previous works 23 . This protocol includes the determination of gains and also some corrections because of cross-coupling effects for specific anatomical angles. The anatomical angles obtained according to the protocol are those shown in Fig. 3.
Data cutting and splitting. The initial and final instants of each record, in which the hands were static, were trimmed. The records were then separated into the different elementary tasks as detailed in the dataset guide by using the labelling performed by the operator while recording the data. In some specific cases in which labelling data was missing, labelling was performed using the video recordings.
Filtering. All data were filtered with a 2 nd order two-way low pass Butterworth filter with a cut-off frequency of 5 Hz.

Data Records
Volume of data collected. A total of 3560 elementary tasks were recorded across all the subjects and experiments, with a total duration of the recordings of 7 h, 30 min and 43 seconds.

112
Carrying utensils and ingredients to the worktop: a bowl, a carton of eggs and a lemon.

113
Carrying ingredients to the worktop: a jar with flour in it, a bag of sugar and a box of baking powder.

114
Carrying utensils and ingredients to the worktop: a carton of milk and a glass.

115
Breaking an egg into a bowl and throwing the eggshell into the bin.

116
Beating the egg with a fork.

117
Filling a glass with sugar.

119
Filling a glass with flour.

120
Opening a carton of milk with scissors and pouring milk.

121
Pouring baking powder into the bowl.

122
Using a mixer to mix the ingredients for the cake dough.

123
Pouring the cake dough onto the baking tray and using a spatula.

124
Putting the baking tray into the oven. Taking the baking tray out of the oven.

125
Cutting a piece of cake with a knife and eating it (simulated).

126
Putting the spatula, the knife, the bowl, the glass and the grater in the sink.

127
Carrying the carton of eggs, the lemon and the carton of milk back to the fridge.

128
Carrying the jar of flour, the bag of sugar and the baking powder to the shelves.

129
Putting the tray with 3 kg of food on it into the oven. Taking the tray out of the oven.

130
Beating an egg and salting it.

131
Preparing the pan for cooking on the hob.

132
Cooking and serving a small omelet.

133
Cooking, serving and cutting a big omelet. www.nature.com/scientificdata www.nature.com/scientificdata/ Data files. Data is presented as a single Matlab data structure (BE_UJI_DATASET.mat) 21 , which is composed of two secondary structures (KINEMATIC_DATA and SUBJECT_DATA). KINEMATIC_DATA contains all kinematic data recorded, classified by experiment, record, part and subject, while SUBJECT_DATA contains data of the subjects recruited (age, gender, laterality, weight, height, hand length, hand width and measured AROM). This structure is accompanied by a guide (.pdf), which provides detailed information regarding the data series as well as the environment, tasks, objects and data acquisition system.  Fig. 4) Notice that movement of thumb CMC joint is complex, and nomenclature used in literature to define these movements is varied 26,27 . We adopted the one used by Brand and Hollister 27 . Table   201 Putting a tablecloth on the table.

202
Placing a dish, a glass, a fork, a knife and a napkin.

203
Placing a jug of water, an oil cruet, a salt-shaker and a bowl.

204
Putting the glass, the jug, the oil cruet and the salt-shaker back in their place.

205
Throwing the leftovers on the plates into the rubbish bin.

206
Throwing the leftovers in the bowls into the rubbish bin.

207
Removing the tablecloth from the table and folding it.

208
Washing the glass, the bowl, the dish, the fork and the knife.

209
Putting the glass, the bowl, the dish, the fork and the knife back in their place.

211
Taking a jar of ground coffee and opening it.

212
Filling the filter handle of the coffee machine with coffee.

213
Placing a cup under the coffee machine and pressing the power button.

214
Placing the cup of coffee and the sugar pot on a tray. Carrying it to the table.

215
Throwing the used ground coffee into the rubbish bin.

216
x Adding sugar to the coffee, stirring and drinking it (simulated).

217
Pouring crisps from a bag into a bowl.

218
Closing the bag of crisps with a sealing clip.

219
Pouring olives from a tin into a little bowl.

220
Pouring salted biscuits from a jar onto a dish.   Table 4. Elementary tasks into which task R = 106 is divided. Columns containing R (ID of the recording), ID (ID of the task), OBJ (ID of the objects used during the task), SCEN (ID of the scenario where the task is performed) and S (marked with an "x" when the task was performed sitting).
www.nature.com/scientificdata www.nature.com/scientificdata/ technical Validation Data acquisition. Before and after carrying out each experiment the subjects were asked to perform movements such as closing their hands or just moving them randomly, in order to make sure that all the gauges were shown to be working on the virtual model of the CyberGlove software.
Furthermore, all tasks recorded were checked in order to ensure that the number of labels used to divide them into elementary tasks was correct and that no labels were missing.
In order to avoid possible unexpected signal values, all data collected were filtered using a 2 nd order two-way low pass Butterworth filter with a cut-off frequency of 5 Hz, as explained in previous sections.
Comparison of active and functional range of motion for each subject and experiment. The percentiles P95 and P5 were calculated for each hand joint, experiment and subject. Then, for each subject and experiment, a subject-specific functional range of motion (FROM) was computed for each hand joint angle as the P5 and P95 percentiles of all his/her recordings, therefore representing the angles of 90% of the postures performed by the subject during the experiment. These FROMs were compared with the AROMs measured for each subject. Almost all the FROMs were inside the AROMs, except in some cases where the extension of thumb interphalangeal and metacarpophalangeal joints and the index metacarpophalangeal joint extension were higher than the AROM measured (maximum difference reported between FROM and AROM was 25° approx.). This may be attributable to activities that implied a passive extension of these joints while manipulating objects (e.g. cutting with a knife implies a precision grasp with a forced extension of the thumb joints and index interphalangeal joint that is higher than the achievable active extension).    www.nature.com/scientificdata www.nature.com/scientificdata/ Statistical descriptive analysis of all data collected. With all data collected, box and whisker graphs were plotted and general FROMs were calculated. Then, the extreme values of all the subjects' AROMs calculated previously were taken to calculate general AROMs. When general FROMs and AROMs were compared, most values of the FROMs were between those of the AROMs, which supports the veracity of the data. Nevertheless, some outliers were higher than those values (Fig. 5), especially in extension of CMC1, MCP2, PIP2 and PIP3 and flexion of right PIP2 to PIP5. This can also be attributable to activities that implied a passive flexion/extension of joints while manipulating, as mentioned before. It has to be emphasized that the FROMs of PIP2 to PIP5 were higher than the AROMs only for the right hand, which is the dominant hand of most subjects.
Limitations. The use of instrumented gloves may imply some loss of dexterity during the performance of fine manipulation tasks. Nevertheless, this loss of dexterity may not have a significant effect on the ranges of motion, mean postures or movement synergies.

Usage Notes
These data can be used for several applications, from machine learning purposes to product design. The main strengths of this dataset for these potential uses are the motion capture characteristics (validity of the motion capture system, anatomical joints measured and frequency of acquisition), the structure of the data presented (.mat, which allows easy data handling), the variety of objects used (different shapes and weights) and the wide range of cooking/feeding tasks considered.
It has to be taken into account that real food or drinks were not used to perform the tasks in order to prevent the gloves from getting stained or wet (all products are appropriately tagged with the corresponding substitutive material in the dataset guide file). Therefore, tasks involving these elements were simulated and might be performed in a slightly different way than when performed with real food/drink.
Even though tasks and products of this dataset were selected to be representative of the different cooking and feeding tasks, some specific tasks or objects involving fine motor skills were discarded because of the loss of manipulation dexterity that the use of instrumented gloves implied (e.g. opening the thermally sealed plastic layer of precooked food packaging). Some wrist angles are also missing because of improper fitting of the wrist sensors to some subjects' hands.
Finally, velocity of performance of the tasks might be slightly affected by the loss of dexterity and touch sensitivity resulting from the use of the instrumented gloves.

code availability
The custom Matlab code used to calculate joint angles is freely available on Zenodo 28 . www.nature.com/scientificdata www.nature.com/scientificdata/ acknowledgements The authors would like to thank all subjects for their participation and Sonia Buralla for her support during the data collecting process and preparation of the material. This work was supported by projects MINECO DPI2014-52095-P and UJI grant PREDOC/2016/08.

author contributions
Alba Roda-Sales created the experimental protocol, acquired data, performed data analysis and wrote the paper. Margarita Vergara conceived and planned the project, conducted and supervised the experiment and reviewed the paper. Joaquín L. Sancho-Bru conceived and planned the project, conducted and supervised the experiment and reviewed the paper. Verónica Gracia-Ibáñez developed the protocol to calibrate the instrumented gloves, developed the Matlab code to calculate joint angles and participated in the creation of the experimental protocol. Néstor Jarque-Bou recruited subjects, helped with the development of the Matlab code to calculate joint angles and revised the data structure.

additional Information
Competing Interests: The authors declare no competing interests.
Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.