does it require every state to agree?

Yes. It’s baked into Article V which is about amendments. The last line is the relevant line.

Article. V.

The Congress, whenever two thirds of both Houses shall deem it necessary, shall propose Amendments to this Constitution, or, on the Application of the Legislatures of two thirds of the several States, shall call a Convention for proposing Amendments, which, in either Case, shall be valid to all Intents and Purposes, as Part of this Constitution, when ratified by the Legislatures of three fourths of the several States, or by Conventions in three fourths thereof, as the one or the other Mode of Ratification may be proposed by the Congress; Provided that no Amendment which may be made prior to the Year One thousand eight hundred and eight shall in any Manner affect the first and fourth Clauses in the Ninth Section of the first Article; and that no State, without its Consent, shall be deprived of its equal Suffrage in the Senate.

I’m not a constitutional scholar, but presumably an amendment cannot self-reference the article that amendments are derived from. Otherwise, we could just amend Article V to remove the last line of text and then amend the Senate as much as we wanted with another.

I could be wrong. Maybe the Founders were hoping that the future generations would notice this, but enough slave owners at the time wouldn’t and sign it.