after images

[scruffy]

Do you get visual after-images when you stare at something for a while?

If so, how long do they last?

There are three theories of after images.

1. they happen in the retina because of the bleaching of photoreceptors

2. they happen in the retina and other areas because when the stimulus goes away, the removal of lateral inhibition causes rebound excitation

3. they happen in the visual cortex, and what happens in the retina is just simple adaptation.

By now we know that both retina and visual cortex are involved. This was determined by a careful analysis of the time course from psychophysics experiments, with the neural firing patterns of individual retinal ganglion cells.

Cortical mechanisms for afterimage formation: evidence from interocular grouping - Scientific Reports

Whether the retinal process alone or retinal and cortical processes jointly determine afterimage (AI) formation has long been debated. Based on the retinal rebound responses, recent work proposes that afterimage signals are exclusively generated in the retina, although later modified by cortical...

www.nature.com

The retina is definitely involved, but the time course of retinal adaptation is either very fast or very slow, and the persistence of afterimages is somewhere in the middle.

Neural Locus Of Color Afterimages - PMC

After fixating on a colored pattern, observers see a similar pattern in complementary colors when the stimulus is removed. Afterimages were important in disproving the theory that visual rays emanate from the eye[1], in demonstrating inter-ocular ...

pmc.ncbi.nlm.nih.gov

Here is my proposal from the standpoint of predictive coding:

The afterimages are perceptual. They can be perceived by closing the eyes (ie "in the absence of sensory input").

After images are persistent error signals. Predictive coding requires that the error signals hang around for a while, and they would behave in the same way as lateral inhibition (in other words, with rebound excitation).

This view also suggests a mechanism for hallucinations, as unmatched error signals.

The sensory visual system can be compared to a Kalman filter, or more specifically an information filter (which is a Kalman variant). Kalman filters are exactly like Bayesian estimators, they take the current state of the system (a priori), make a prediction of the next state (likelihood), and use it to calculate a difference (called error, based on an energy function or cost function), which is then used to update the filter parameters (a posteriori).

Kalman filters make the Markov assumption, which is "no memory", only the current state (and the input) is used to calculate the next state. They are a form of "hidden Markov process", which conveniently lets us add noise to any term (for analytic or generative purposes).

The Kalman filter has a time scale. Let us say the system state is sampled at time t1, and updated at time t2. Then you can look at the update as occurring at (t1+t2)/2, and r1 and t2 being equidistant from the origin. Let us imagine that these times now move farther away from each other, away from the origin in the direction of + and - infinity. Consider these scenarios:

1. You can stagger multiple Kalman filters.

2. You can increase or decrease the cycle time.

3. You can have all n Kalman filters connected together to pick up the similarities and differences in the model at each point in time. In this way you get a picture, of both the system states and the predictions.

In the information filter specifically, you can run multiple window spans through the same filter. Why this matters:

Norm (mathematics) - Wikipedia

en.wikipedia.org

This in turn affects downstream calculations like entropy.

So, when you're minimizing errors using the squared sum of residuals, this is the same as minimizing the cross entropy which is the same as minimizing the KL divergence (which is the distance between the actual and predicted distributions).

 

[scruffy]

These pictures are from the cerebral cortex of the human brain.

Here is one neuron, with 5400 of its inputs. Fibers are in blue, synapses are in green.

Unseen details of human brain structure revealed

A detailed map of all the cells and connections in a single cubic millimeter of the human brain revealed previously unseen aspects of brain structure.

www.nih.gov

I'll assume that you're familiar with the early visual system. The short story is retina => LGN => V1, where V1 is the primary visual cortex located at the top of the occipital pole in back of the head. Neurons in retina and LGN have round receptive fields, they respond to dots - but neurons in V1 have elongated receptive fields like bars, they respond to edges at particular orientations and angles.

So here you are, you're a neuron in V1. The predictive coding scenario goes like this:

1. A bar of light falls on the retina, that matches your receptive field. The hard wired connections in V1 detect that "aha! This is an edge at a 45 degree angle to the horizontal". (In other brain areas this might be a lookup in an associative memory, but in V1 all the connections become hard wired shortly after birth).

2. You make a prediction about the LGN neurons feeding you. You say "I predict this one and that one are on, but these other ones are off". This is based on the orientation of the stimulus edge, so your prediction is going to be matched with a bunch of 0's from all the other neighboring neurons (the ones representing different edge orientations, that don't match in this case). This is "sparse coding", it's very efficient.

3. Your prediction is stored, and now you're going to compare the current input with the prediction you just made. But in the meantime, a fly lands on one of the retinal neurons and turns it off. So now you have an error. So every one of the LGN cells is going to return "no error" to you (meaning your prediction is correct), except for that one cell where the fly.landed. So now you have to evaluate the error.

4. You evaluate the error and generate a new prediction. After the error, do you still believe it's an edge at 45 degrees or do you now believe it's something else? After doing your Bayesian math, you decide the chances are good it's still an edge, so that's the prediction you send down.

5. Just the the fly takes off, so now you have a perfect prediction and you get a 0 error signal from the LGN.

There is a physical thermodynamic law driving this behavior, it's called the Free Energy Principle.

Free energy principle - Wikipedia

en.wikipedia.org

Let's quickly consider V2, the next highest visual center after V1. Maybe it says "that edge is part of a table", and when the fly lands and V1 is kvetching about whether the edge is still an edge, there is actually no change in the error signal it sends to V2. It keeps telling V2 "I have an edge" even when the fly lands on it. So the representation of "table" stays consistent even while it's going through gyrations with edges and flies.

There is OTHER processing that let's V2 determine "the edge of the table has a fly on it". It involves a parallel visual pathway from the retina to the superior colliculus. It matches the position of the dot with any predictions containing it. This subsystem is involved with visual attention, and it contains a number of reflexes including species specific survival mechanisms (like the looming reflex in rodents, which we've talked about before).

All of this related to visual perception. Unattended stimuli are not "perceived" even when they fall directly on the retina. In normal operation attention is divided so you allocate a small piece of it to processing sensory input, and usually it's the novel or meaningful things that catch your eye. In extreme cases more attention can be devoted to a single stimulus or it's features. If you're worried about mosquitos carrying West Nile Fever, you might want to check out that fly, so you'll have to notice it.

The weird thing about after images is they can persist for many seconds, long after your retinal photoreceptors have recovered. If you're staring at a bright pink dot and then you have to look at the fly, you might not see it at first but you will after you blink. However if you close your eyes at that point you'll see the after image of the pink dot

 

[scruffy]

Afterimages have some interesting properties. First, this will show you the relevance of the graphs above:

Cortical shape adaptation transforms a circle into a hexagon: a novel afterimage illusion - PubMed

After viewing a colored figure on a uniform gray background, an observer will see a negative afterimage after the colored figure disappears. This study shows that the shapes of afterimages vary systematically according to the shape of the adaptation stimuli, a phenomenon that could be caused...

pubmed.ncbi.nlm.nih.gov

Second, there is interocular transfer. Looking at a bright image with one eye closed will sometimes cause the afterimage to transfer to the closed eye.

Cortical mechanisms for afterimage formation: evidence from interocular grouping - PMC

Whether the retinal process alone or retinal and cortical processes jointly determine afterimage (AI) formation has long been debated. Based on the retinal rebound responses, recent work proposes that afterimage signals are exclusively generated in ...

pmc.ncbi.nlm.nih.gov

Third, there's a lot to be learned from visual illusions. Our brains can generate afterimages even from illusory surfaces, proving central involvement.

https://www.science.org/content/article/illusions-wont-go-away#:~:text=Stare%20at%20some%20sights%20long,is%20responsible%20for%20the%20trick.

 

[scruffy]

The condition called palinopsia looks like this:

Illusory palinopsia - Wikipedia

en.wikipedia.org

You are perceiving something that isn't really there. Very close to hallucination.

Interestingly enough, the cerebellum gets involved in visual processing. There are 5 independent topographic maps of visual space in the cerebellum. Cerebellum coordinates eye movements, among other things. Normally you'd expect a "former" visual signal to be reset after an eye movement, yes?

 

[scruffy]

Someone can make a quick name for themselves in the machine learning field by modeling this:

That's an eye and its neural targets.

You see four chemical gradients (proteins called ephrins), and from these four gradients we generate retinotopic maps with single cell precision.

Each eye covers the entire visual field, but splits the field at the optic chiasm. Neurons handling the left side of the visual field go one way, neurons handling the right side go the other.

There is one symmetry breaking event driving this organization. Early in development, a stem cell destined to become a retina divides in two along the nasal-temporal axis, and each cell remembers which side it is. The Fox proteins get involved in this, and thereafter nasals express one way and temporals express another.

https://www.tandfonline.com/doi/full/10.1177/1759091414562107#d1e748

Your task is to self organize this network using only the four chemical gradients and the nasal-temporal identity of each cell.

You'll need 1 million retinal ganglion cells per eye. (Chump change for modern networks). Fortunately, the synaptic divergence per cell is only about 5 target neurons in each direction.

For extra credit, generate two separate maps from the same retina and then get them to align with each other.

 

[scruffy]

Here's a fascinating read.

The lesson from machine learning is everything is based on statistics.

The view I'm advancing in this thread is that after images are evidence of the nature of the brain's statistical processing.

As a guiding principle then, the statistics of creatures in their natural environments would be a driving factor for evolution.

The big success of machine learning is token (object) identification and classification. In evolution though, this takes on a secondary significance.

Primary in evolution, are things like hunting and escape, and reproduction. Our early visual systems were built to handle these imperatives.

Here is the story of the organization of reflexive visual behaviors in the superior colliculus of the brain. It enables the organism to track and eat anything that moves, and avoid threats from above. The amazing convolutional layers in the cerebral cortex come much later in evolution, they handle the case of "no, don't eat it, it's poisonous".

https://www.sciencedirect.com:5037/science/article/pii/S0959438823000983

 

[scruffy]

The interesting thing about afterimages is they're perceived.

They're not sensory, they're purely internal. "Subjective", as it were.

In keeping with the predictive coding hypothesis, experience is associated with neural feedback pathways, rather than the feed forward pathways.

Experiments have shown that birds have sensory awareness. Crows in particular, can be trained to perform in psychophysical experiments where they report their subjective perceptions. This capability depends on the nidopallium, which is the avian equivalent of the frontal cortex.

Crows can be trained to peck one lever if they see red, and a different lever if they see green. (Crows are tetrachromats, they see color even better than humans). In a fixation task involving a bright red stimulus, the crow will press the red lever till the stimulus disappears, then press the green lever till the afterimage disappears.

https://royalsocietypublishing.org/rstb/article/380/1939/20240308/235178/Conscious-birdsConscious-birds

 

[scruffy]

Hopefully you've all seen Scruffy's explanation of the Hawaiian earring and why it's important for awareness and consciousness.

What makes a predictive neural network work, is a concept very similar to a Kalman filter.

This is a Kalman filter tracking a moving object in 3 dimensions:

A neural network performs the same calculations as a Kalman filter. It combines a prediction of the next state with the current sensory input, and updates the internal model based on any errors.

The brain (at least the cerebral cortex) is a collection of 100 million Kalman filters that are operating asynchronously. You can calculate approximately how many filters are updated in any given time slice, and what you find is the internal model changes smoothly when the data is consistent, and abruptly when it's not.

In addition, each filter is connected to every other filter. So basically a single filter that's being updated, is getting its input from a window of time depending on when the other filters are updated. This interconnection of all the filters into one big network is called the Global Workspace. Many of the other brain networks (like the attention networks) regulate what gets into the Global Workspace. You "see" something when the signal hits your visual cortex, but you only "understand" it when it enters your global workspace.

Quite obviously, the global workspace is coordinated across the whole brain, and there are multiple brain regions regulating this activity. One of the important ones (one of many) is the superior frontal gyrus, which contains the supplementary areas that feed the frontal eye fields (Brodmann area 8, extending through 9 and possibly into 10):

This area is involved in self awareness (among other things), and it connects with just about every other area of the cerebral cortex. It gets input from the 3d spatial map in the parietal lobe and the scene map in the hippocampus, and it connects with the attention system in the anterior cingulate cortex. This area "drives" the attention system, in other words the voluntary selection of things to pay attention to. It's lateralized, with spatial working memory and language on the left, and behavioral inhibition and impulse control on the right (in right handed people). Attention is also linked with arousal, vigilance, and emotion, and these functions are closer to area 9 while area 8 is closer to spatial working memory.

Again, the architecture is consistent - a set of brainstem nuclei for orienting and simple targeting, an analytic system containing a "what" part and a "where" part that get combined into a scene map, and top down attentional control that dynamically interacts with the entire sensorimotor hierarchy to keep everything smooth and focused.

Each Kalman filter learns the trajectories of all the others. They all interact in the Global Workspace. At any given point in time an update gets information from a window of representation (prediction) time centered on "now". It takes a small amount of time for oculomotor commands to get from the filters to the targeting areas, and we can track this activity by looking at the oculomotor premotor potentials in the cortex and brainstem.

This is just the red area and below, that turns the eyes, neck, and head to target a visual or auditory stimulus.

Premotor potentials are a big discussion, we'll save it for some other time