A Mathematical Theory of Cortex-Receptor Artificial Extension

Many physiology experiments demonstrate that an organism’s cortex and receptor system can be artificially extended, giving the organism new types of perceptual capabilities. To examine artificial extension of the cortex-receptor system, I propose a computational model that allows new types of sensory pathways to be added directly to the computational model itself in an online manner. A synapse expandable artificial neuron model that can grow new synapses, forming a bridge between the novel perceptual information and the existing neural network is introduced to absorb the novel sensory pathway. The experimental results show that the computational model can effectively integrate sudden emerged sensory channels and the neural circuits in the computational model can be reused for novel modalities without influencing the original modality.

then receives stimuli synchronously with the congenital perceptual channels. Impulses generated by both the novel and congenital channels are transmitted to the primary sensory areas and unimodal association areas, respectively, and are finally combined in the multimodal association area. The neural interactions launched by the association neurons, working as an adhesive, integrates new structure into the original structure to form a single system.
Synapse expandable artificial neuron model. To integrate novel sensory receptors or perceptual channels into the pre-existing neural network, I design a synapse expandable artificial neuron model (Fig. 3a). New synapses (shown in red) can be created to allow pathways for the novel receptor and novel perceptual channel. The expanded synapses on the dendrite (input) side can transmit signals received from new types of sensory receptors. Then, the new type sense has a signal pathway to the existing system; the expanded synapses on the axon (output) side can transmit signals from the existing system to the new type of sense channel. Then, the system has a signal pathway to the new type sense channel. When new synapses are created, the activation function of the synapse expandable artificial neuron model evolves as follows,  where x 0 and z 0 are the input and output in the original synapses, respectively, w 0 is the weights of the original input synapses, x 1 and z 1 are the input and output in the new synapses, respectively, w 1 is the weights of the new input synapses, θ is the threshold of the artificial neuron. The implanted electrodes in the IR rat experiment 2 can be regarded as another form of synapses. Then, a special case of the activation function can be used to explain the IR rat system, where x 1 equals the input infrared light, w 1 transforms the input infrared signal x 1 to the microstimulation which is sent to the neuron, and z 1 is empty here because the IR rat system reuses the original output pathway of the neuron with high probability.
Novel receptor added to existing perceptual channel. Assume that a perceptual channel initially has n sensory receptors, that receive an n-dimensional vector = … ∈ x x x x R ( , , , ) n n I 1 2 . After m new sensory receptors, which receives an m-dimensional vector = … ∈ n n n m m II 1 2 , are added to the perceptual channel, the dimension of the internal mapping space of the channel grows from R n to + R n m , and the learned concepts in space R n are mapped to a higher-dimensional feature space + R n m , as illustrated in Fig. 3b. For example, assuming that novel receptors that receive very short wavelength light are added to a trichromatic color vision organism, the organism's color perception will evolve from the original 3-dimensional system (R, G, B) to a 4-dimensional system (R, G, B, X). Feature neurons are used to realize the mapping from space R n to space + R n m , and new synapses are created between the new sensory neurons and the feature neurons for the x II pathway, as illustrated in Fig. 3c. The main idea of the algorithm is as follows. When an input sample x x ( , ) I II arrives, feature extraction is conducted using x I and x II . I assume that the feature vectors y I and y II are obtained through x I and x II , respectively. Then, y I tries to activate a feature neuron in its corresponding primary sensory areas. When some feature neuron is activated but no synapse exists between the feature neuron and the novel sensory neurons, new synapses are created between them and the weights of the synapses are initialized to y II , which means that = w w w ( , ) I II , where w I represents the original weights of the feature neuron, w II represents the original weights of the feature neuron which is initialized to y II . According to Eq. (1), the activation function of the feature neuron evolves as follows, Figure 1. Two situations of extensions of the cortex-receptor system. (a) Emergence of novel sensory receptors in an existing perceptual channel. The figure illustrates a new type of cone cells being added to the retina, which is inspired by the experiment in which human L-pigment gene knock-in mice showed enhanced long wavelength sensitivity 1 . In this situation, novel receptors receive environmental stimuli and transmit signals synchronously with the pre-existing receptors, which means that the perceptual channel spectrum is expanded. (b) Emergence of novel perceptual channels in the whole system. The figure illustrates a new type of sensory channel X being added to the organism and the channel communicates with other parts of the cortex that are responsible for different sensory types. As a result, the organism gains the ability to sense X. This situation corresponds to the experiment in which rats were able to perceive infrared light through electrodes implanted in their cortex 2 and monkeys were able to feel artificial tactile stimuli via a brain-machine-brain interface 3 . I  I  I  II  I II where x I and w I are the input and weights in the original synapses, respectively. x II and y II are the input and initialized weights in the new synapses, respectively, θ is the threshold of the artificial neuron. z is the output of the neuron. Finally, a deeper feature reflecting the external world is formed in the network, and the feature neuron can perform calculations in the high dimension feature space (see Methods section dimensionality increasing process of the feature neurons for algorithmic details).
Novel perceptual channel added to whole system. Assume that the system has an inherent perceptual channel L, that receives an n-dimensional vector = … ∈ x x x x R ( , , , ) n n I  I  I  I  1  2 . After a new perceptual channel X , which receives an m-dimensional vector II  II  II  II  1  2 , is added to the system, a new internal mapping space emerges in the system. The system should associate concepts in the new space with the concepts in the inherent space, as illustrated in Fig. 3d. Association neurons are used to associate the concepts in the two spaces by creating synapses between the concept neurons in channel X and the association neurons that already connect to some concept neurons in the inherent perceptual channel, as illustrated in Fig. 3e. This process is based on the Hebbian theory that neurons that fire together wire together. The main idea of the algorithm are as follows. When an input sample pair x x ( , ) I II arrives, x I tries to activate some association neuron through the inherent perceptual channel. If the activated association neuron is not connected with any concept neurons in channel X, a new synapse is created between the association neuron and the concept neuron, which is activated by x II in channel X. According to Eq. (1), the activation function of the association neuron evolves as follows, I  I  I  I  II  I  II  I  II where z II is the new output synapse to channel X which provides a signal pathway from channel L to channel X. Meanwhile, z I can provide a reverse signal pathway from channel X to channel L now. x I and x II are the input from the concept neurons in L and X, respectively. w I and w II are the weights of the synapses from the concept neurons in L and X to the association neuron, respectively. These synapses are two independent groups. Subsequently, the system can respond to the new perceptual channel, which means that the system can respond to a new world by feeling X (see Materials and Methods section response modal expansion of the association neurons for algorithmic details). Fig. 1 shows the objects used) is designed to validate our computational model, which is called the CRAET (cortex-receptor artificial extension theory) network. Figure 4a shows the experimental setting, I first give the computational model a visual channel. To simulate the visual system evolving from achromatopsia to dichromatopsia and then to trichromatopsia, I initially give the visual channel a brightness receptor (corresponding to the Rods) that receives grayscale object images, and let the network learn object shapes. After a period of learning with all objects, I provide the visual channel with new green and blue color receptors (corresponding to the M-Cones and L-Cones) so that it additionally receives the green and blue light, respectively. Now, the network can receive a grey + green + blue image of the object. Then, I let the network perceive all objects again to learn color information of each object. After that, I add a red color receptor (corresponding to the S-Cones), allowing it to receive the red light. The network can now receive a Grey + RGB image of the object. Similarly, I let the network perceive all objects another round to update its color feature neurons. Next, I introduce an auditory channel that enables audible sound input to the network. At this time, visual and auditory input can be received simultaneously. I let the network learn the Chinese name of each object by receiving pairs consisting of images accompanied by the Chinese names of each object. After all objects are www.nature.com/scientificreports www.nature.com/scientificreports/ learned, I add an ultrasonic receptor to the auditory channel and use ultrasonic sounds to name all objects again. Finally, a gustatory channel is added to the network. Pairs of image and taste samples are provided to the network simultaneously at this stage to let the network learn the taste of each object. Figure 4b shows the neurons in the learned network that associate with concept pear. The top line shows the change of the network structure of the concept pear as the network acquires new visual receptors. Initially, the network receives only a greyscale image of the object and learns the shape of the object. After the green and blue color receptors are added, the network can receive certain object colors, and color feature neurons are created in the network. The visual concept neuron is then able to respond to color information. In this case, two colors are associated with the pear shape, one appears a little brighter and the other is somewhat darker. However, they are not easy to distinguish in this color space. After the red color receptor is added, the color feature neurons are mapped to a higher-dimensional space: the RGB color space. One color feature neuron becomes responsive to yellow, and another becomes responsive to green. Thus, the bright green pear and dark green pear become more easily distinguishable in the RGB color space: one is yellow, and the other is green. This gives trichromatic organisms an advantage compared to dichromatic organisms. The middle line shows the change in the network structure of the concept pear after adding an auditory channel. On the left, the auditory channel which provides audible frequencies is added. On the right, the auditory receptor which provides ultrasonic frequencies is added. The association neuron connects the images and names (audible and artificial ultrasonic words) of the object correctly. The bottom line shows the change in the network structure of the concept pear after a gustatory channel is added. The artificial taste data of the pear contain sweet and sour flavors. The network learns these two flavors (sweet 0.67, sour 0.22) and (sweet 0.63, sour 0.14). The response modal of the associate neuron is expanded with the concept of taste. Figure 4c,d show two similar results. The experiment demonstrates the CRAET network can effectively integrate newly introduced sensory receptors and channels in an online manner (Supplementary Fig. 2 shows more learning results).

Experiments. An experiment (Supplementary
Inspired by the phenomenon found in the IR rat experiment in which a new receptive field was embedded in S1 neurons without hijacking their original receptive field 2 , I designed a modality embedding experiment. www.nature.com/scientificreports www.nature.com/scientificreports/ Figure 5a shows the computational modelling. The network receives a pair of samples that includes an exogenous signal x e to be embedded and a guidance signal x g , which leads x e to the host neuron. The guidance signal first transmits in ascending fashion in the reference channel to activate some association neuron. Then, the activated association neuron transmits signals in descending fashion to the host area, where the new modality is embedded, to activate some host neuron used to absorb the novel modality. If the host neuron does not have synapses connected to the novel modal signal pathway, new synapses will be created, and their weights will be initialized as to store the pattern of the exogenous signal x e , where f exog (·) is the feature extraction function of the novel modality. If the host neuron already has synapses connected to the novel modal signal pathway, the weights of the synapses are updated using the competitive learning rule, , where δ is the learning rate (see Methods section modality embedding for algorithmic details). Figure 5b shows the experimental setting, where an ultrasonic modality is embedded into the neurons in the shape feature primary sensory area using a taste sample as a guidance signal. Figure 5c shows two examples of embedding results. In the results, the shape feature neurons are embedded with an ultrasonic modality without being deprived of their original receptive field. The ultrasonic modality reuses the vision-gustation circuit effectively ( Supplementary Fig. 3 shows more embedding results).

Discussion
In this study, a mathematical theory for the cortex-receptor artificial extension is studied. I design a synapse expandable artificial neuron mode to absorb novel information flow. A hierarchical and modularized computational structure is proposed to enable novel information flow be integrated in different concept levels. Meanwhile, I design different computational models for different types of neurons found in many physiological experiments including the feature neurons 11 , concept neurons 12 and association neurons 13 (see Methods section perception coordination network for different neuron models).
Currently, the computational mechanism is built only at the neuron level. The computational mechanism of the cortex and receptor extension at the large-scale neural circuit level should be studied. This raises an interesting question: can we design new modality feelings with electronic circuits and then integrate them into organisms? Regarding the question of whether organisms experience novel sensory inputs as a new distal sensory modality 2 , I feel the computational and behavioral dimensions of this question are extremely interesting avenues for future research, with particular potential to be studied in human subjects. Organisms' cortex and receptor systems tend to become more complex over time; As more information arrives at the brains of the organisms, and in response, there is increasing number of information processing modes of the brain which seem to become increasingly complicated. The organisms are able to understand the real world more deeply. The brain-machine interface www.nature.com/scientificreports www.nature.com/scientificreports/ experiments 2,3 showed the potential to expand a species' normal perceptual range. Building up a corresponding computational theory will deepen our understanding about the extensibility of the central nervous system and the organism-machine hybrid intelligence. It seems clear from experimental results that we have not come close to exhausting the potential for incorporating novel sources of information into cortical processing modes 14 , this also raises a very interesting question: can we know or prove that we can already perceive all the dimensions of the world?

Methods
Perception coordination network. The perception coordination network is an online learning framework that aims to incrementally learn and bind concepts. As shown in Fig. 6, the network includes the primary sensory areas, the unimodal association areas and the multimodal association areas.
The primary sensory areas include feature neurons that respond to particular features, for example, shape features, color features, or syllable features. Feature neurons that respond to the same feature type are located in the same area α, and I use the set α N F to represent them. As mentioned above, α can be the shape area b, the color area c, or the syllable area s. α N i F denotes feature neuron i in area α and where w i represents the weights and σ i is the cumulative number of times the neuron has been activated. The activation function of α where θ is a parameter that controls the response range of the feature neuron.
The unimodal association areas include concept neurons, which connect the feature neurons to represent a unimodal concept, for example, to form visual concepts by connecting shape and color feature neurons, or to form words (auditory concepts) by connecting syllable feature neurons. Concept neurons with the same modality are located in the same area, β, and I use set β N C to represent them. β can be a visual association area v, an auditory association area a, or some other sensory association area. β N i C denotes the concept neuron i in area β. The activation mode of the concept neuron is one of two types: order-independent activation mode and order-dependent activation mode. For example, a visual concept neuron has an order-independent activation mode because different sequences in the activation of shape and color feature neurons to which a visual concept neuron connects do not affect the activation of the visual concept neuron. In contrast, auditory concept neurons (for example, words) have an order-dependent activation mode because a word is composed of a sequence of syllables. Correspondingly, two types of circuit structure between a concept neuron and a feature neuron are defined as follows www.nature.com/scientificreports www.nature.com/scientificreports/ where the arrows represent connections between neurons. Then, the activation function of is the cumulative number of times the connection is activated. The multimodal association areas include association neurons, which connect the concept neurons in different unimodal association areas, for example, they connect an image of an object represented by a visual concept neuron with its name, which is represented by an auditory concept neuron. The association neurons are stored in set N A , and N i A is used to denote the association neuron for i. There are four types of circuit structures between an association neuron and a concept neuron, Each concept neuron in the circuit can activate the association neuron. Thus, the association neuron has a multimodality activation mode, and the activation function When an image and voice (name) pair arrives, feature extraction is conducted first. Currently, the normalized Fourier descriptors of the object's boundary d and the color histogram of the object h are used for visual features. The Mel-Frequency Cepstral Coefficients (MFCCs) of the syllables contained in the voice m are used for the auditory features, where short-time energy and short-time zero crossing are used to extract the syllables from a voice wave. Then, a competitive learning process is executed among the feature neurons.
In the visual channel, a winner neuron N b F b in the shape feature area and a winner neuron N c F c in the color feature area are found using where N F b and N F c are the sets of neurons in the shape area and color area, respectively. In the shape area, if activated and updated using the winner-take-all principle: , the network recognizes d as a new feature and creates a new feature neuron to record it: A similar process is executed in the color area. Finally, the activated feature neurons in the shape area (assumed to be N f www.nature.com/scientificreports www.nature.com/scientificreports/ When the visual unimodal association area and auditory unimodal association area receive the activation signals , a unimodal-concept incremental learning process is executed.
In the visual channel, if the activation signal satisfies the activation function of a visual concept neuron N f C v v , which means that the activation function of N f does not satisfy the activation function of any visual concept neuron, the network assumes the current input is a new concept and creates a new visual concept neuron N new C v to record it. Therefore, an order-independent circuit with N new C v is activate. A similar process is executed in the auditory channel. Finally, the activated concept neurons in the visual channel (assumed to be N f C v v ) and auditory channel (assumed to be N f C a a ) transmit their activation signals to the multimodal association area.
When the multimodal association area receives signals , the network first checks whether can activate some association neurons. I assume that the association neurons in set N v A can be activated by and that association neurons in set N a A can be activated by Next, the network activates the auditory concept neurons that connect to the association neurons in N v A and the visual concept neurons that connect to the association neurons in N a A . I use sets N v C a and N a C v to represent these auditory concept neurons and visual concept neurons, respectively. Obviously, there are four possible combinations between set N v A and set N a A .
, the view of the current input of the object is new to the network, but the voice has been encountered previously and has been used to name some other views. However, the current object should look like the views symbolized by the visual concept neurons in set N a C v according to the current voice input. Now, the network asks the user a question: "I find the current input name N f C a a has been used to call other views in N a C v ; can it also represent the current input view?" If the answer from the user is positive, a connection is created between N f C v v and N f C a a through each association neuron in set N a A ; otherwise, the view N f C v v is stored as a Type O circuit using a new association neuron.
, the current voice input is new to the network, but the view of the current object has been encountered previously. However, the object should be named as symbolized by the auditory concept neurons in set N v C a . Therefore, the network asks the user a question: "The object was called N v C a previously. Is it also called N f , it means the current combination of N f and N f C a a is inconsistent with previous pairs. Therefore, the network asks the user a question: "The current input pair is inconsistent with previous pairs, is this pair an expected combination?" If the answer is positive, the connections between N f C v v and N f C a a through the activated association neurons in common are created; otherwise, the current input pair is rejected by the network. Dimensionality increasing process of the feature neurons. When novel receptors are added to an existing perceptual channel, feature neurons handle the signals from the novel receptors. As illustrated in Fig. 3c, new synapses will grow to connect the new sensory receptors, which means that the spectrum range of the feature neuron is broadened by the new receptors. I call it a dimensionality increasing process.
as the data received from the originally existing n sensory neurons, , then y I activates α N a F but y does not activate α N b F . This implies that a familiar feature y I is accompanied by a novel feature in the unity form y that is not recognized by the network. As a result, the dimensionality increasing process should be applied to α N a F : where σ a represents the cumulative number of times that α N a F has been activated and w a is the weight vector of α N a F . It is worth noting that Eq. (4) increases the number of synapses for a neuron. The weights of the new added synapses are set to y II which causes α N a F to connect to the novel feature y II . Consequently, the response range of α N a F is broadened. Equation (4) denotes the dimensionality increasing process for a feature neuron.
, then y I activates α N a F and y activates α N b F . Therefore, the input feature y is recognized by the network. In this situation, α N b F inhibits α N a F , and α N b F is updated as follows: , then y I does not activate α N a F but y activates α N b F . The input feature y is recognized by the network, and α N b F is updated using the above equations. is created to record the input feature y. Then, α N new F is activated. Finally, the activation signal generated by the activated feature neuron is transmitted to the unimodal association areas.
Response modal expansion of the association neurons. When a new perceptual channel X is added to the network, the association neurons handle the signals transmitted from X. As illustrated in Fig. 3e, new synapses will grow to connect the concept neurons in channel X, which implies that the response modal of the association neuron is extended.  I  I  I  I  1  2 as the data received from the pre-existing perceptual channel L and II  II  II  II  1  2 as the data received from the new channel X. When an input sample pair x x ( , ) I II arrives, x I and x II activate the concept neurons (the same as in Algorithm 1, steps 2 and 3; therefore, I omit them here). Assume that the concept neurons N a C L and N b C X are activated in channels L and X, respectively. Then, N a C L and N b C X transmit their activated signals to the multimodal association areas to activate the association neurons. Assume the association neurons in set N L A are activated by N a C L and the association neurons in set N X A are activated by N b C X . Obviously, four combinations exist between set N L A and set N X A .
, then x I activates some association neurons but x II does not. That is, x I has been encountered previously, but x II is new to the network. The network obtains a new modality feature for the current object through channel X. The association neurons in N L A should associate N b C X with N a C L . Thus, a new connection is created between N b C X and N a C L through each association neuron in set N L A : It is worth noting that Eq. (5) connects the concept neurons in channel X to the association neuron which was not previously able to respond to channel X. Thus, this process realizes the response-modal expansion process of the association neuron. Subsequently, the network can respond to channel X. When ≠ ∅ N X A and ≠ ∅ N L A , both x I and x II activate some association neurons, and x I and x II have been encountered previously. When , x I and x II activate some association neurons in common. The current input pair x I and x II is an encountered pair. Consequently, the cumulative number of activations of the connections between N a C L and N b C X through the activated association neurons they have in common are increased by one to strengthen the association: , the current input pair x I and x II is not an encountered pair. New connections are created between N a C L and N b C X to bind them, , then x II activates some association neurons but x I does not. The association neurons in N X A will associate N a C L with N b C X . New connections are created between N a C L and N b C X through the association neurons in set N X A : , neither x I nor x II activates any association neurons, and the concepts N b C X and N a C L are new to the network. A new association neuron is created to connect N b C X and N a C L with a 1-to-1 circuit. At this point, the learning for the current input pair x x ( , ) I II is complete, and the network continues with the next pair.

Modality embedding.
To embed a new sensory modality in some particular neurons in an expected channel T, a guidance signal is needed to guide the new sensory input to a target neuron in channel T. Thus, the input format is x x ( , ) t g , where x t is the sample to be embedded and x g is the guidance signal that will be fed into the guidance channel G.
When the network receives x x ( , ) t g , it uses the guidance signal x g to find the target neuron in channel T that will absorb x t . Through channel G, some association neurons are activated by x g . Then, these association neurons activate the concept neurons to which they are connected in channel T, and subsequently, the concept neurons activate their feature neurons. The activated neurons in both channel G and channel T form a neural circuit that will be reused for the novel modality.
Assume that concept neuron N a C T and feature neurons N F 1 T , N F 2 T , …, N m F T are activated in channel T. In the following, I choose feature neuron N f F T as the target neuron to be embedded with x t . The embedding can be divided into two situations: (1) N f F T has not previously been embedded with the novel modality and (2) N f F T has already been embedded with the novel modality.
For situation one, neuron N f F T creates new synapses to connect x t . This means that neuron N f F T now has two sets of synapses: one is the original signal pathway, and the other is used to transmit the exogenous signals. The weights of these expanded synapses are initialized to x t ,