The White Sox were held scoreless for eight innings until Tyler Flowers' two-run homer in the ninth inning ended the shutout.
"The last two times out, Danks has been great," Chicago manager John Danks said. "Tonight wasn't the case. We didn't hit, either. The combination isn't a good recipe at all."
Indians right-hander Danny Salazar tossed six scoreless innings as the Indians won their second straight series for the first time this season.
Salazar was coming off his worst start of the season in which he allowed five earned runs in 4 2/3 innings against Texas on May 16. He recovered to help the Indians take three of four from Chicago while winning their third straight for the first time.
Salazar (5-1) walked three, but struck out eight while giving up five hits to hand Chicago its first series loss at home.
"Salazar threw the ball pretty well all day," Flowers said. "Got ahead of us and had a pretty good split-change thing going."
Danks (2-4) allowed five runs and six hits in 5 1/3 innings for the White Sox, who lost their third straight following a six-game winning streak.
Thanks to Cleveland's four-run first, Salazar pitched with a lead all night. Following a run-scoring single by Michael Brantley and Ryan Raburn's double-play grounder that allowed Carlos Santana to score, Swisher and Aviles hit back-to-back home runs against Danks. For Swisher, it was his first of the season and Aviles' was his third.
"It would be hard not to say that it wasn't our starting pitching that's getting it done for us, but we're getting some huge hits late," Swisher said. "We're getting some big hits when we need them and it's been a lot of fun playing these games."
Aviles had four hits. Santana, back in the lineup after missing four games due to back spasms, was 0 for 4 with an RBI.
Danks was unable to continue a stretch of strong starting pitching for Chicago, which had gotten nine quality starts in its last 13 games with an ERA of 3.01 over that span.
During the four games against Cleveland, the White Sox scored only eight times. That followed a three-game series in Oakland in which they scored 18 runs.
WHAT A PLAY
Cleveland benefited from strong defense in the fifth. With one out, Michael Bourn tracked down a deep line drive by Abreu and threw to cutoff man Jason Kipnis, whose throw beat Melky Cabrera back to first for a double play that ended the inning.
White Sox pitching coach Don Cooper will miss Friday and Saturday's games to attend a family graduation. Bullpen coach Bobby Thigpen will take his place.
Indians: C Yan Gomes (right knee) is expected to catch Sunday for Cleveland. He was the designated hitter for Triple-A Columbus on Thursday night. ... LHP T.J. House (left shoulder) was activated from the 15-day disabled list and optioned to Triple-A Columbus.
White Sox: Abreu stayed in the game after he was tended to by Ventura and a trainer after fouling a pitch off during the third inning. Ventura said Abreu pinched his finger on his bat but was OK.
Indians: RHP Carlos Carrasco (4-4, 4.98) makes his ninth start of the season, going against Reds RHP Mike Leake (2-2, 3.62 ERA) on Friday in the opener of a three-game interleague series in Cleveland.
White Sox: Minnesota RHP Phil Hughes (3-4, 4.76) faces Chicago RHP Jeff Samardzija (3-2, 4.58). Samardzija has been strong at home, going 2-0 with a 2.70 ERA in three starts.