There were several factors in both the Soviet Union and United States that contributed.
1. Khrushchev was an unpredictable leader. He was more open and willing to work with the West than Stalin, but at other times was a hard line old school communist (the invasion of Hungary, Cuban Missile Crisis, etc). Though Khrushchev was more willing to work with the West than Brezhnev, at least the latter brought a certain amount of stability to the table. The West knew what to expect from him.
2. By extension of above, the simple stasis that the Soviet Union had reached under Brezhnev. Life wasn't great, but it was tolerable. Basically everyone living in a major urban area in the USSR had access to food (much of it imported from the USA), a kitchen, indoor plumbing and any of the other necessities required for bare mediocre life by the time of detente. At this same time, information and travel were restricted making everything relatively stable. Brezhnev realized this and knew the country's problems with balancing the budget and maintaining an equal technology level with the West and thus chose to simply keep with the status quo rather than potentially rock the boat.
3. The rise of Communist China as a competitor rather than an ally to the Soviet Union put the Soviets on the defensive as the leaders of World Communism. The USSR became more worried about maintaining influence in their own backyard rather than far abroad where they were more likely to butt heads with the U.S.
4. The United States was somewhat reeling from the Vietnam War and no President was willing to escalate any major conflicts.
5. Civil Rights, Social Welfare and Environmental causes became more important to the U.S. in this period. This not only meant that the U.S. was more focused on domestic issues, but that the Soviets lost a lot of their argument about the capitalist West being so abusive to their population and not caring about their own societal welfare. I don't know if you are familiar with the old Soviet Tu quoque argument they often used "And you are lynching Negroes!", well once these kind of blatant disregards for human and civil rights became much less commonplace, a lot of the Soviet argument in this area was gone. Of course the Soviets would continue to lie to their population about what was going on in the West, but information did inevitably leak into the Soviet Union and damage was done to the image of the West as being so evil.
6. Economic decline in the United States. Detente happened basically the same time as oil prices were rising and as Japanese auto manufacturers were becoming true competitors in the U.S. market. This meant not only that everyone was more focused on domestic issues, but the U.S. share of global trade declined as a whole.
7. Arms control treaties. The Cuban Missile Crisis wasn't good for either side. Both the Soviet Union and United States saw the real risk of nuclear war. This combined with the fact the Soviet Union was pretty much poor, the United States was more focused inward and the rise of new (and unpredictable) nuclear powers such as China and India meant that both sides now had incentives to control arms. First amongst these was the NPT (Nuclear Proliferation Treaty) which also led to the SALT talks. New technologies such as ABMS (Anti-Ballistic Missile Systems) were scary to both sides as they encouraged a first strike and lead to the ABM Treaty.
