Blog

In this post, you will learn how to solve the “ValueError: Index has duplicate keys” error in Pandas.

The full text of the error is:

Traceback (most recent call last): File “C:\Users\panda\PycharmProjects\venv\my_file.py”, line 9, in my_df.set_index([‘Id’], drop=True, inplace=True, verify_integrity=True) File “C:\Users\panda\PycharmProjects\venv\lib\site-packages\pandas\util\_decorators.py”, line 331, in wrapper return func(*args, **kwargs) File “C:\Users\panda\PycharmProjects\venv\lib\site-packages\pandas\core\frame.py”, line 6068, in set_index raise ValueError(f”Index has duplicate keys: {duplicates}”) ValueError: Index has duplicate keys: Index([‘1003′], dtype=’object’, name=’Id’) Process finished with exit code 1

I get an error while creating an index in the dataframe.

How to resolve ValueError: Index has duplicate keys error in Pandas

There are two ways to solve this problem and below I will discuss two different solutions to get rid of the ValueError: Index has duplicate keys error.

Remove duplicate indexes

The first way to fix the error is to get rid of duplicate indexes. This method is dedicated to people who want the index to be unique.

In order to remove duplicate indexes you can get rid of duplicates in your dataframe or change the value of duplicate indexes if they were entered incorrectly.

Disable the verify_integrity option

The second way is to disable the verify_integrity option by setting this parameter to False, which is the default value. This solution is dedicated to people who do not care that the values in the index column are unique.

For example, I am pasting the command that will be used to trigger the ValueError error. In this case, change the verify_integrity parameter value to False or remove it.

my_df.set_index([‘Id’], drop=True, inplace=True, verify_integrity=True)

See also:
Documentation of set_index method