monitoring and managing SQL replication

Dear all,

The new company I am going to join has replication problem and it is slow and they don't know why, how can I:

1) monitor the efficient of the replication ?
2) any network connective requirement on setting up the replication ? they might fail those requirement.
3) how to change the replication type so that we can minimize the replication traffic?
4) any built in function/XEvent to keep track of any lost in replication during the whole day so we can see the pattern ?
5) usually what is the way to enhance it/adjust the efficient of the replication,
6) any built in report feature of SQL server so that I can see how many lost of replication, how many record replication conflict last night ?
LVL 1
marrowyungSenior Technical architecture (Data)Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

arnoldCommented:
There are many possibilities, the first of which is to determine what they currently have.
What is the available bandwidth, how active is their DB meaning how much data and at what frequency ....
Before you change anything you have to know what their business process/requirements are to make sure the change you are contemplating while might improve in one category, might become detrimental in another.

MS guide to monitoring replication.
https://msdn.microsoft.com/en-us/library/ms147874%28v=sql.105%29.aspx

Enhancing replication performance guide
https://msdn.microsoft.com/en-us/library/ms151786%28v=sql.105%29.aspx


Since you are starting with a new firm, do not rush into making changes until you are familiar with what is going on and test, verify and test and verify before attempting a change.

Hopefully the person who set it up, is there and whose brain you can pick to determine what the considerations were at the time this was setup.
0
marrowyungSenior Technical architecture (Data)Author Commented:
"There are many possibilities, the first of which is to determine what they currently have.
 What is the available bandwidth, how active is their DB meaning how much data and at what frequency ...."

tks, I will find that out once I onboard  and get back to this question.

"Before you change anything you have to know what their business process/requirements are to make sure the change you are contemplating while might improve in one category, might become detrimental in another."

I can list out the function it can serve and let them make decision !

"Hopefully the person who set it up, is there and whose brain you can pick to determine what the considerations were at the time this was setup. "

he is gone! that's why the new company keep asking if I can come onboard earlier!

all the SP here: https://msdn.microsoft.com/en-us/library/ms147874%28v=sql.105%29.aspx only available when replication setup ?

one thing, I have script to verify how many record lag behind the SQL mirroring for each user database, any similar script to monitor the status of replication? e.g. how many record is missed and how many in the queue?

it seems that this link : https://msdn.microsoft.com/en-us/library/ms151786%28v=sql.105%29.aspx do not show information about SQL server 2012/2014, any reason for this? all function moved to DAG group of SQL server 2012/2014.



The replication is good in the way that we can make decision, down to table level, what item we can replication ? mirror and log shipping you know is about the WHOLE database.
0
arnoldCommented:
Use the first link monitoring replication, change to the SQL version . Navigate out to operations, there you will see pergormance.......


You likely can not use mirroring/log shipping because the DBS are accessible on multiple sites. Read and write.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

marrowyungSenior Technical architecture (Data)Author Commented:
"
 You likely can not use mirroring/log shipping because the DBS are accessible on multiple sites. Read and write. "

yeah for sure, but my question on this is: "The replication is good in the way that we can make decision, down to table level, what item we can replication ?"

I didn't use any replication at all for all my previous job, surprise ! only mirror (DR) and log shipping (read only )

when I execute this "exec sp_replmonitorhelppublisher;"

my local empty SQL server 2014 returns:

"Msg 208, Level 16, State 1, Procedure sp_replmonitorhelppublisher, Line 49
Invalid object name 'msdb..MSdistributiondbs'.
"

why ? because there are no replication setup at all ?

so there are no single script to monitoring the replication but SP one bye one ?
0
marrowyungSenior Technical architecture (Data)Author Commented:
any requirement on the network connectivity ? URL for this if you have!

where SQL server store the log/error message/ error log of the existing replication option so that I can find it out easily ?
0
arnoldCommented:
replication should be triggered by a job, job history will be an indication when/how frequently it runs and the results, there is the error log within SQL and application eventlog.  

Each of us has a process how I approach a problem.
You might fill your tool bet with tools that will not fit the issue.
I.e. when one has a hammer, everything appears to be a nail.
The first thing I would do is to familiarize what applications are being used and how. Then track it back to to the application/system for which you are responsible.  Once you can define what the issue is, then you start working your way back from the application/system of which you are in charge to determine whether it can be corrected from your side.
one always hopes the step they take improves or does no harm, the no harm is most likely. The rare issue is that while familiarizing oneself with the environment, the person who knew about the application  x/Functionality that is rarely accessed/used but is automated forgot to mention it, you make an adjustments and a week, two, a month go by and you have an irate person in front of you that a report that was supposed to arrive today has not.

You were hired because of your skill set, make sure you understand completely what is going on before you try to put your stamp on the situation.
Bandwidth is informational.  I.e. you know how much data has to be replicated, and you have to make sure  that the connections they have can allow for such transfer in an orderly manner.

lets say they have a GB worth of changes per day, their connections are asynchronous 5Mb download and 768kb upload.
their replication is setup to run every 15 minutes. The maximum transfer rate is 768Kbits, the slower of the two.

making the replication run on  a more frequent basis may improve one thing while having an additional load on the system.

At least from my point of view, that would be my approach.

make sure to backup the replication plans/publicher/subscribers,etc. before you do anything.
Usually, replications that break for more than a certain amount auto-delete/terminate.  if you do not have those backed up to recreate them, .........world of hurt.
0
marrowyungSenior Technical architecture (Data)Author Commented:
"replication should be triggered by a job, job history will be an indication when/how frequently it runs and the results, there is the error log within SQL and application eventlog.  "

ok, surprise to see replication is not going to keep going without any job, agree?

"The first thing I would do is to familiarize what applications are being used and how. Then track it back to to the application/system for which you are responsible."

they will have a training for me.

"Once you can define what the issue is, then you start working your way back from the application/system of which you are in charge to determine whether it can be corrected from your side."

the dBA gone and they has a plan to let me check what is going on and what is the suggestion, so I have to find a place to quickly check out what is happening and happen, at this moment, they just said replication is slow. so the first thing from my mind is network connection .

so for you, the quickiest way to look at the problem is for sure the SQL log and windows log? so there are no replication log of the da y ?

"The rare issue is that while familiarizing oneself with the environment, the person who knew about the application  x/Functionality that is rarely accessed/used but is automated forgot to mention it, you make an adjustments and a week, two, a month go by and you have an irate person in front of you that a report that was supposed to arrive today has not."

yeah, I knew, so question should I ask is :

1) how frequent is the replication ?
2) what talbes will be replicate and what is the schedule
3) what time they exeperinece delay or all the time ?
4) any incident like records are missing after replication ?

right? what is your suggestion ? never touch replication before.

"lets say they have a GB worth of changes per day, "
right and good, any metric in SQL server to show me how many data has been replicated ?

"their replication is setup to run every 15 minutes. "

so replication is Async mode/method, right?

"make sure to backup the replication plans/publicher/subscribers,etc. before you do anything."

any step by step on how to back it up ?

"Usually, replications that break for more than a certain amount auto-delete/terminate."

sorry ,what is that mean ?
0
arnoldCommented:
The set thing first, if a replication setup breaks meaning normal replication can not proceed, instead if continuing building up log files, the replication setup auto terminates, the job, subscribers,publishers are canceled or at least that happened once, a server configured as publisher on a "retiring system" experience HW failure, was not noticed for a couple of days, when server booted back up. All the replication related jobs, .. Were gone.

If memory .., using ssms, locating the replication publisher/subscribers you can export the configuration I.e. The command to create it if needed..

The reviewing replication settings will answer the questions you post what db, table/s criteria, and type
The replication/performance are a part of the entire document that could help you familiarize your self with the answers you are seeking.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
marrowyungSenior Technical architecture (Data)Author Commented:
ok,

terminates you keep saying is the role, right?

"The set thing first, "

I can't setup on my laptop as there are no other SQL server there and no much to see for empty database, right?

so I can only check what has been setup in the new company and then check out performance problem, and that's why I ask how to see performance by log,etc.

" the replication setup auto terminates, the job, subscribers,publishers are canceled or at least that happened once, a server configured as publisher on a "retiring system" experience HW failure, was not noticed for a couple of days, when server booted back up. All the replication related jobs, .. Were gone.
"

I don't know undertstand why is it ! why all jobs gone as the server already booted back up?

"If memory .., using ssms, locating the replication publisher/subscribers you can export the configuration I.e. The command to create it if needed.."

can script out you mean ?

"The reviewing replication settings will answer the questions you post what db, table/s criteria, and type "

how to view the replication setting ? is one of the question I am asking.

"The replication/performance are a part of the entire document that could help you familiarize your self with the answers you are seeking."

you mean this ?

"MS guide to monitoring replication.
https://msdn.microsoft.com/en-us/library/ms147874%28v=sql.105%29.aspx

Enhancing replication performance guide
https://msdn.microsoft.com/en-us/library/ms151786%28v=sql.105%29.aspx"
0
arnoldCommented:
Yes,
Root of the replication documentation.

https://msdn.microsoft.com/en-us/library/ms151198(v=sql.105).aspx

Setup two SQL instances on your laptop laptop\primary laptop\sevondary

And you an test configure what you need between them.

Using ssms, I  think there is a portion deals with displaying publishers and subscribers.

Has been a while.
0
marrowyungSenior Technical architecture (Data)Author Commented:
"Using ssms, I  think there is a portion deals with displaying publishers and subscribers."

sorry I don't understand what it means, please explain again.
0
arnoldCommented:
when you open Sql Server Management studio and connect to your database server, there is a Repication folder with local subscriber/local publishers.
Right clicking on the one you have and ..........
0
marrowyungSenior Technical architecture (Data)Author Commented:
"when you open Sql Server Management studio and connect to your database server, there is a Repication folder with local subscriber/local publishers."

yeah! got it now , I am guessing that too.... :):):

I will hold this ticket for a week as I will go to work on Monday and my first week will take a look on some replication stuffs and will get back on what I found.

"making the replication run on  a more frequent basis may improve one thing while having an additional load on the system."

one thing, do you think SSMS provide good monitoring console to monitor the replication ? only from SSMS can monitor everything? no other tools is needed ?
0
arnoldCommented:
monitoring scripts jobs notification and a course of action......
0
marrowyungSenior Technical architecture (Data)Author Commented:
""monitoring scripts jobs notification "

you are talking about using the replication SQL job notification's alert message to monitor it?
0
arnoldCommented:
To notify you when the job encounters a failure.
Monitoring is one thing, but since it is unlikely that you will be watching and checking, the alert, notification so that the issue can be identified and corrected.
0
marrowyungSenior Technical architecture (Data)Author Commented:
"Monitoring is one thing, but since it is unlikely that you will be watching and checking, the alert, notification so that the issue can be identified and corrected. "

ok, let the email tells then.. I will then.

but by this sense, it seems that when I come onboard tomorrow,  I can take a look on the job history for the first round of replication health checking ?

this is one of the check of all SQL jobs, agree?

bTW, any replication health checking tools you can suggest ?
0
arnoldCommented:
Look at many things, get an idea on the type of replication, how the application the users use functions, look at logs, look at the destination (subscriber/publisher)'s ....

First thing figure out what is going on before jumping to and try to fix when the complete picture is understood to a point.
0
marrowyungSenior Technical architecture (Data)Author Commented:
sorry, on the new job and they don't allow me to see what is going on in the replication and I don't have sysadmin right at all on the production.

will go to US weeks later and they will train me for everything.

once I can see the replication I will come back for this optic.

"First thing figure out what is going on before jumping to and try to fix when the complete picture is understood to a point."

of course, I don't have any right to see anything related to replication at this stage.

"how the application the users use functions"

what is that mean ?

" look at logs, look at the destination (subscriber/publisher)'s ....
"

look at all their logs ?
0
arnoldCommented:
SQLite is where the data what is being used to view/access/write .....
0
marrowyungSenior Technical architecture (Data)Author Commented:
I am sorry, SQLite is the SW to monitor the replication you mean ?
0
arnoldCommented:
Autocorrection replaced what typed and did not notice the change.

What application do the user's use to access the data and how it interacts.  Do they have multiple location with each location reading/writing to their local SQL instance with transactional/merge replication the data is consolidated from each to the others.
0
marrowyungSenior Technical architecture (Data)Author Commented:
"Autocorrection replaced what typed and did not notice the change."

ok, you mean what you typed, right?

" Do they have multiple location with each location reading/writing to their local SQL instance with transactional/merge replication the data is consolidated from each to the others. "

this is what replication used to be/used. we are not going to use log shipping/mirroring for this as all office has change to the data itself and replication will merge the change and solve the conflict, right?

so what SQLlite is about ? ignore it as this is the autocorrect make it wrong ?

"What application do the user's use to access the data and how it interacts."

you mean the first is to understand which table need replication, right?

" look at logs, look at the destination (subscriber/publisher)'s ....
 "

 look at all their logs ?
0
arnoldCommented:
Certain typos, grammatical errors are all my doing.  Could not remember what I wanted to say, nor could I refresh my memory reviewing your prior comment.

The application the users use is to get an understanding ....

Thus far, what information have you been able learn about the setup? Or does it all have to wait until after your training in SQL?
0
marrowyungSenior Technical architecture (Data)Author Commented:
my company currently just give me sysadmin role on the publisher and I will take a look on it more.  I will go to US and during that time I might have more and more question on this but for now let me close this question first, please come back to this question if you see I post again

the picture will be much clear 2-3 weeks later while I am in US.
0
marrowyungSenior Technical architecture (Data)Author Commented:
yeah, I found that both replication and log shipping user SQL job as the trigger.
0
marrowyungSenior Technical architecture (Data)Author Commented:
but one thing, is replication also make use of log shipping as well?

when I open any replication/distribution job, it shows:

"d:\Program Files\Microsoft SQL Server\100\Tools\Binn\sqllogship.exe" -Restore 1C96Etr9890-7E77-459C-B76C-85EryuD164 -server YYYY

at this moment email alert about replication is only these:
1) REPLICATION ALERT Unresponsive Agent for publication xxxRep1 on subscriber YYY. Subscriber running for 107 min
2) REPLICATION ALERT Unresponsive Agent for publication Documents on subscriber YYY. Subscriber running for 19337 minutes.
3) REPLICATION ALERT FailedAgent for publication xxxRep1 on subscriber YYY. Subscriber running for 12 minutes.
4)REPLICATION ALERT Unresponsive Agent for publication Accounting on subscriber ZZZ. Subscriber running for 118 minutes.
5)REPLICATION ALERT Unresponsive Agent for publication Domestic on subscriber ZZZ. Subscriber running for 118 minutes.
6)REPLICATION ALERT Unresponsive Agent for publication Shipping on subscriber ZZZ. Subscriber running for 118 minutes.
7)REPLICATION ALERT FailedAgent for publication Purchase_Order on subscriber ZZZ. Subscriber running for 22 minutes.


so there are not much information can be dig into, right? it just who is subscriber and no more, right? I receive this information everyday.

why so much unresponsive agent ?
0
arnoldCommented:
The replication uses the transactions record during the time interval to update the the subscriber from the publisher. Replication also uses special/table/columns ......
The difference between replication and log shipping, is that in replication the databases on each side are accessible (read and write) while in log shipping setup only one server is accessible (read/write) while the other is always in recovery mode to which the transferred transaction log is applied (restored)

Look at the message (full message headers) to identify the sending server if not identifiable from the message (from/subject)

Once identified, look at the configuration of the publisher/subscriber create a map of your replication environment which server to which server, Name of publication/subscription and which tables are impacted......

Failed would suggest an issue.
Unresponsive could be that load might be an issue.
0
marrowyungSenior Technical architecture (Data)Author Commented:
"The difference between replication and log shipping, is that in replication the databases on each side are accessible (read and write) while in log shipping setup only one server is accessible (read/write)"

yeah knew it.

"while the other is always in recovery mode to which the transferred transaction log is applied (restored)
"

the log shipping model the secondary server is in restore mode for read only operation, right?

"Look at the message (full message headers) to identify the sending server if not identifiable from the message (from/subject)
"

sending server means publisher? what we need to distributor for?

and this kind of replication also use log shipping method to send information from publisher to subscriber ?
0
marrowyungSenior Technical architecture (Data)Author Commented:
seems script on detecting the publisher, distributor and subscriber, right?
0
arnoldCommented:
publisher, subscriber, distributor each could be a separate systems and each can generate events.  Looking at the message headers will tell you from which system the event was sent.  

Database in recovery mode means it can not be accessed read nor write, the restore of a transaction log backup is a write (append of recent changes or new additions) that is run by a job into .... (I think the state of the DB would be restoring/in recovery.....
0
marrowyungSenior Technical architecture (Data)Author Commented:
"Database in recovery mode means it can not be accessed read nor write"

it should be in standby/read-only mode.

so replication also use log shipping method to send information from publisher to subscriber ?
0
arnoldCommented:
You can have a publisher which is where the data is, the subscriber which is where the data needs to go, a distributor is the one handles the transfer/initiation.....
Crude, best look at https://msdn.microsoft.com/en-us/library/ms152567%28v=sql.105%29.aspx
0
marrowyungSenior Technical architecture (Data)Author Commented:
0
arnoldCommented:
A database that is in recovery mode, the only changes that can be done to it is an addition of SQL transaction logs.
It can not be accessed by any user.

Log shipping is a specific term.  
The link I posted covers the replication, a follow on link included there in explains the replication agents .....

log shipping, primary live server, has three jobs on a set interval it generates a transactional log backup and stores it in a directory that is shared. The server/s to which the data needs to be restored are the "notified" each has their own three jobs, copy the transaction log file from the share, and restore it to the local Database with no recovery mode meaning leave the database in the recovery mode..

Depending on which replication is in place, transaction from one are "transferred" to the other, the means by which it is done are different......
0
marrowyungSenior Technical architecture (Data)Author Commented:
tks, reading this :

https://msdn.microsoft.com/en-us/library/ms152567(v=sql.100).aspx it takes me some times to read.

"A database that is in recovery mode, the only changes that can be done to it is an addition of SQL transaction logs.
 It can not be accessed by any user"

so it restore log backup, this is how log shipping works ! that's what I mean and feel about replication too !

"Log shipping is a specific term.  
 The link I posted covers the replication, a follow on link included there in explains the replication agents ....."

tks, will read

"log shipping, primary live server, has three jobs on a set interval it generates a transactional log backup and stores it in a directory that is shared. The server/s to which the data needs to be restored are the "notified" each has their own three jobs,"

that's what are we having, will primary log shipping server has diff number of jobs than log shipping secondary ? each only have 3 ?  

hey which 3 x of both server , primary : 1) backup 2) save ?  secondary for sure is 1) copy from primary 2) restore and what is 3) ?
0
arnoldCommented:
If memory serves, one job deals with notification.

Monitoring job status should be done no matter whether it is a log shipping, replication, backup, reorganize/rebuild index, etc.

Job.

The replication handles the incorporation of the changes differently than a log shipping setup.
0
marrowyungSenior Technical architecture (Data)Author Commented:
"The replication handles the incorporation of the changes differently than a log shipping setup. "

tks, this is good enough.

then the server I am touching is a log shipping secondary server.

"If memory serves, one job deals with notification."

I think that job do this : exec sys.sp_check_log_shipping_monitor_alert  ?

and I found that if there are more than one log shipping secondary DB, they share the same alert job.
0
marrowyungSenior Technical architecture (Data)Author Commented:
Dear all,

just get more information from the existing DBA in US on this problem:

Biggest problems with replication are that is has a lot of errors. Some errors cause reinitialization. With the network lag time from Vxxx to Axxxx, this can mean many hours of down time.

One of the biggest problems is that tables must be created at the publisher

any idea on this ? that's why I keep asking network latency requirement on this , should be more on network problem.

also any other method to replace replication by other means? tools and software can be suggest ! DAG group ? or fix the network first?
0
arnoldCommented:
To fix network issues, you need to know how much data you are dealing with flowing through.
The issue you have multiple sites. Replication provides localized access to data that us then consolidated .......

The SQL traffic/interaction is likely high and that is the reason for this setup.otherwise a geographic cluster with or this would gave been considered I.e. Everybody accesses the cluster.......
0
marrowyungSenior Technical architecture (Data)Author Commented:
"To fix network issues, you need to know how much data you are dealing with flowing through."

so for this we have to check with network guys I think. or how can we measure it.

" Replication provides localized access to data that us then consolidated .......
"

you mean all data put on distributor so make data access easier ? then we should focus on movie closer to the subscriber ?
0
arnoldCommented:
You have if memory serves 10-12 sites each with their own SQL server in merge/transactional replication?data from 9-11 has to make to the one and .......

What is the existing.  What is the frequency at which the replication jobs run......

I understand the reason behind the inquiry, but until you get the info on the setup and related training, you might be going down a path ....
0
marrowyungSenior Technical architecture (Data)Author Commented:
I will check it in US next week.
0
marrowyungSenior Technical architecture (Data)Author Commented:
frequency is about 30 minutes and our Richmond,, VA's DBA say it is network problem.
0
arnoldCommented:
depending on the load, 30 minutes might be too long. A shorter interval means you will have less data to transfer but you will have more frequent merge/update.... Check whether every 15 minute processing is an option.

IF the DBA state site says it is network related, check with network to see if there is available capacity.
0
marrowyungSenior Technical architecture (Data)Author Commented:
yeah. but we are going to do it next year. new network need cost!
0
marrowyungSenior Technical architecture (Data)Author Commented:
basically has no idea on how large the data is  going to transfer.

we just tried to copy a large file like 1GB from publisher to subscriber, we have network guys here watching that the file seems never complete the  transfer and  getting slower and slower. it is transferring but very slow if transfer time is long.

"publisher, subscriber, distributor each could be a separate systems and each can generate events.  Looking at the message headers will tell you from which system the event was sent.  "

you mean by checking application log?

also now I am not sure it is using merge replication or transaction replication, how to know it ?
0
arnoldCommented:
Checking the message headers of the email notification about the status/failure. It should reflect the source server that sent it.
What is the bandwidth utilization?
Is the transfer over a VPN, is there a limit on the VPN rate/bandwidth/prioritization?
0
marrowyungSenior Technical architecture (Data)Author Commented:
"Is the transfer over a VPN, is there a limit on the VPN rate/bandwidth/prioritization? "

It is not VPN, it just across the internet !

We will change to dedicated network between SQL server next year.

"What is the bandwidth utilization?"

I just know we have little bandwidth now.

we are using merge replication.

the message header will not show you which one is publisher, which one is the distributor (we might not have it), and who is the subscriber, right? or it will by default  ?
0
arnoldCommented:
The message that sends out the alerts is the starting point on a treasure hunt.
Once you can identify the monitoring/sql agent that generates the notification, you can find the next step i.e. is this the distributor, the publisher, the agent or a combination of two publisher/distributor distributor/subscriber.

You have to have a starting point.  Message headers is the best way to determine the source of the email.
A system that is being used as the mail server, will also include the reference on the server from which it got the message

Received: From sending server
   by receiving server
Received: from "pickup folder"
   by sending server
From:
To:
Subject:
Messasge-ID: <fsdfgsdf>

message
               
If you have multiple sites with merge replication, without network/server relationship/topology/references, the only thing faster is to use the notifications. Otherwise, you have to go server by server documenting what is on it (i.e. since you are on the server, might as well document as it might come in handy later on. Meaning something on this server might be used, but is not apparent....)

I understand your wish to be up and running, your position is more of a long distance runner versus a sprinter.

Pace ourself.
0
marrowyungSenior Technical architecture (Data)Author Commented:
we will have this fixed soon, we are receiving message header.
0
marrowyungSenior Technical architecture (Data)Author Commented:
there are reason you deserver my score man..
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SQL Server 2008

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.