CARTAGENA, Colombia — President Barack Obama's diplomatic mission to Latin America this weekend threatened to be overshadowed by alleged misconduct by a dozen Secret Service agents sent to provide security for him in Colombia.

On Friday night, a caller who said he had knowledge of the situation, told The Associated Press the misconduct involved prostitutes in Cartagena, site of the Summit of the Americas. A Secret Service spokesman did not dispute that.

The White House had no comment, but also did not dispute the allegations.

A U.S. official, who was not authorized to speak publicly on the matter and requested anonymity, put the number of agents at 12. The agency was not releasing the number of personnel involved.

The alleged activities took place before Obama arrived Friday in this Colombian port city for meetings with 33 other regional leaders. Secret Service spokesman Ed Donovan said the agents involved were relieved from duty and replaced with other agency personnel.

"These personnel changes will not affect the comprehensive security plan that has been prepared in advance of the president's trip," Donovan said.

Still, the allegations were an embarrassment for the president and his delegation while guests of the Colombian government. And the incident threatened to torpedo White House efforts to keep the president's trip focused squarely on the economy and boosting U.S. trade ties with fast-growing Latin America.

Obama was to hold two days of summit meetings with regional leaders before heading back to Washington Sunday night.

The agents at the center of the allegations had stayed at Cartagena's Hotel Caribe. Several members of the White House staff and press corps were also staying at the hotel.

A hotel employee, speaking on the condition of anonymity for fear of losing his job, said the agents arrived at the beachfront hotel about a week ago. The employee described the agents as drinking heavily during their stay.

The employee said the agents left the hotel Thursday, a day before Obama and other regional leaders arrived for the weekend summit.

The hotel's public relations chief had no comment.

The Washington Post reported that Jon Adler, president of the Federal Law Enforcement Officers Association, said the accusations related to at least one agent having involvement with prostitutes in Cartagena. The association represents federal law enforcement officers, including the Secret Service.

Adler later told the AP that he had heard that there were allegations of prostitution, but he had no specific knowledge of any wrongdoing.

Donovan said the agency personnel involved had been sent back from Colombia to their permanent place of duty. The matter was turned over to the agency's Office of Professional Responsibility, which handles the agency's internal affairs.