You have some conflict with theme or other plugin. Please insert this in Custom wpDataTables CSS under the “Custom JS AND CSS” in main settings of wpdatatables.

.wpdt-c .modal{ width: inherit !important ; background-color: initial !important; }