What are shards

With respect to behaviors: A shard is a contextually activated chain of behaviors which terminate in the acquisition of reward. A subshard is a part of this contextually activated sequence.

Example: An infant randomly crying sometimes gets reinforced with suger-on-tongue; seeing the red disk appear in vision and then reaching for it also gets reinforced with sugar-on-tongue. The prior is reinforced because it results in the latter. Together, these comprise a shard. The “reaching | see lollipop” piece of the chain is a subshard.

Questions I have about this definition:

• Are parallel paths to same reward the same shard? I can imagine other causes which would get us to “lollipop in field of vision”. Would we call them the same shard (since, terminally, they are reinforced with the same reward) or would we call them different shards?
• Is the shard vs. subshard distinction arbitrary? Due to the infinite redux of causal chains, it seems the initial cause which constitutes the beginning of a shard can stretch back arbitrarily far. Therefore, having an absolute threshold for which to define a shard seems hard.
  • However, relative definitions are easy, any chain can be grouped as a shard, and then subshards are just subsequences of the that reference chain.

Development of shards

A lot of actions are reinforced at each reward event, but the noise quickly dies when the baby samples the actions which were just noise, and does not get the reward they expect.

Agentic shards

Some shards are rote while others are agentic. The division between them is blurry, whereby the main qualitative difference with agentic shards is that they increase the likelihood that the inputs which activate them. E.g.

• Clearly agentic shard: A massive “prioritize education” shard, which may comprise millions of subshards which are rewarded for pursuing education, will try to not let the human be exposed to situations where they may lose the opportunity to learn (e.g. work a 9-to-9 job).
  • Although I don’t see what a “prioritize education” shard would look like relative to the smaller shards I wrote about earlier.
• A shard that, by this given definition, is agentic, but feels like it should be rote: A shard which, upon seeing a glass of wine, immediately consumes it.
  • This will result in the person becoming addicted to alcohol and purchasing more wine, resulting in more situations where they receive this visual input.
• A rote shard: A problem-solving shard which applies a particular algorithm immediately upon seeing a problem type they recognize. This certainly does not increase the frequency that that problem type comes up.

Negotiation between shards

Shards are in a zero-sum competition for access to the input features, because each time one shard gets reinforced, the same features become more likely to feed into that shard as opposed to all of the other shards.

Agentic shards–shards which I now define as those which try to increase the frequency at which cognitive inputs trigger a subsequence within them–can thus come into conflict with each other, because all of them want to increase the probabilities that a certain set of cognitive inputs trigger their sequence.

Thus, these shards want to prevent other shards which are mutually exclusive to them to be reinforced. This results in a natural pressure which would destroy the current human’s values. E.g.:

• Taking drugs feel like a bad idea even though it will bring a lot of joy, because it would get reinforced so much that none of the current cognitive patterns that the human has have voting power anymore. Thus, all current cognitive patterns protest together
• Going to a tech-heavy university as someone who cares deeply about humanistic causes feels bad, because the humanistic shard is worried that the tech/earn money shard would get strongly reinforced.

This would be a mysterious behavior if we modeled humans as value optimizers. If my value function was fixed regarding humanistic causes, then my decision on whether to go to said tech-heavy school or to a humanities-focused school which is less involved in the present/future would depend solely on which was more instrumentally useful for my ultimate value function, but that is clearly not how I think. Instead, I think in terms of “will doing X preserve my values”, not “will doing X maximize my long-run utility function”.