Apparently - at least, up to Monarch - it will research everything, even the optional stuff, and that seems to be a major reason I get/maintain a tech lead.
The Civ3-AI will (eventually) research every tech at
all difficulty levels, if it has time to do so!
The main reason there's a difference in tech-pace between the levels basically boils down to its handicaps or bonuses (AI cost-factor), because that's what governs how
quickly it can (build/capture new towns to) do or buy all that research.
And remember, (even while it might sometimes not feel like that !) the Civ3-AI is purely reactive: it doesn't plan ahead
at all. It chooses its next research-project based on what prerequisite techs are currently known to it, and what the 3-4
potential projects will do in and of themselves. And all the AI-tribes on a map are using the same tech-valuation routine, so will generally follow roughly the same tech-priorities, with only minor/rare variations (based on e.g. UUs, and also AFAIK some degree of pRNG-influence).
The Civ3-AI generally 'prefers' to research techs which unlock new Wonders, buildings, units, governments and/or abilities, the more the merrier. Therefore, "empty" (or at least,
emptier) techs represent the bottlenecks; so as a general rule, if you are at tech-parity with an AI-Civ, then researching the empty(est) 'next tech' as fast as you can, is more likely to give you trade-bait (if you want it) — or to open up a new monopoly-tech beeline that the AI won't get to until it's finished running through the shiniest techs on its
current beeline.
That's a(nother) reason why the Industrial era is frequently the turning-point where the human draws level/ pulls ahead even in higher-level games. By ignoring optional Medieval techs (including the expensive Democracy-beeline), the human gets to the Industrial that much faster than the AI-tribes (all other factors being equal) — either while they're still finishing the Medieval optionals, or while they're still locked into the upper-branch 'Nationalism' upper-beeline.
That latter happens because Nationalism gives so many 'advantages' (new unit, new treaty-options, new governor-button), that the AI generally chooses it over Steam (new resource, new Worker-job), and
always over Medicine (empty) — which will generally be the last of the 3 first-tier techs researched (exceptions generally only occur when Sci-Civs 'luck out' and get Med as their freebie, allowing them to trade for Steam and/or Nat).
And once an AI-Civ has Nat, it can then see Communism (new Gov! New building! New Specialist! Ooh, shiny!), and Fascism (new Gov!), so it will be more likely to go for those, before even 'considering' the rest of the tree.